Information processing device and non-transitory computer readable medium

ABSTRACT

An information processing device includes a controller that controls a notification of a linkage function which is executable by using at least one object to be linked when a user transmits information of the object to be linked to a conversation partner on an interface for making a conversation with the conversation partner.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on and claims priority under 35 USC 119 fromJapanese Patent Application No. 2017-174219 filed Sep. 11, 2017.

BACKGROUND Technical Field

The present invention relates to an information processing device and anon-transitory computer readable medium.

SUMMARY

According to an aspect of the invention, an information processingdevice includes a controller that controls a notification of a linkagefunction which is executable by using at least one object to be linkedwhen a user transmits information of the object to be linked to aconversation partner on an interface for making a conversation with theconversation partner.

BRIEF DESCRIPTION OF THE DRAWINGS

Exemplary embodiments of the present invention will be described indetail based on the following figures, wherein:

FIG. 1 is a block diagram illustrating a configuration of a devicesystem according to a first exemplary embodiment of the invention;

FIG. 2 is a block diagram illustrating a configuration of a terminaldevice;

FIG. 3 is a block diagram illustrating a configuration of an imageforming device;

FIG. 4 is a block diagram illustrating a configuration of a server;

FIG. 5 is a schematic view illustrating the external appearance of theimage forming device;

FIG. 6 is a view illustrating a device function management table;

FIG. 7 is a view illustrating a linkage function management table;

FIG. 8 is a view illustrating a screen;

FIG. 9 is a view illustrating a screen;

FIG. 10 is a view illustrating a screen;

FIG. 11 is a view illustrating a screen;

FIG. 12 is a sequence diagram illustrating an operation of the devicesystem;

FIG. 13 is a view illustrating a screen;

FIGS. 14A and 14B are views illustrating composite images;

FIG. 15 is a view illustrating a linkage function management tableaccording to Modification 3;

FIG. 16 is a view illustrating a screen;

FIG. 17 is a view illustrating a screen;

FIG. 18 is a view illustrating a device function management tableaccording to Modification 4;

FIGS. 19A and 19B are views illustrating device images;

FIGS. 20A and 20B are views illustrating device images;

FIG. 21 is a view illustrating a screen;

FIG. 22 is a view illustrating a screen;

FIG. 23 is a view illustrating another example of the device functionmanagement table according to Modification 4;

FIG. 24 is a view illustrating a linkage function management tableaccording to Modification 4;

FIG. 25 is a view illustrating a screen;

FIG. 26 is a view illustrating a linkage function management tableaccording to a second exemplary embodiment;

FIG. 27 is a view illustrating a screen;

FIG. 28 is a view illustrating a screen;

FIG. 29 is a view illustrating a screen;

FIG. 30 is a view illustrating a linkage function management tableaccording to a third exemplary embodiment;

FIG. 31 is a view illustrating a screen;

FIG. 32 is a view illustrating a screen;

FIG. 33 is a view illustrating a linkage function management tableaccording to a fourth exemplary embodiment;

FIG. 34 is a view illustrating a screen;

FIG. 35 is a view illustrating a screen;

FIG. 36 is a view illustrating a screen;

FIG. 37 is a view illustrating a screen;

FIG. 38 is a view illustrating a screen;

FIGS. 39A to 39E are views illustrating images associated with aconversation partner;

FIGS. 40A to 40E are views illustrating images associated with aconversation partner;

FIG. 41 is a view illustrating a screen;

FIG. 42 is a view illustrating a screen;

FIG. 43 is a view illustrating a screen;

FIG. 44 is a view illustrating a screen; and

FIG. 45 is a view illustrating a screen.

DETAILED DESCRIPTION First Exemplary Embodiment

Descriptions will be made on a device system which is an informationprocessing system according to a first exemplary embodiment of theinvention, with reference to FIG. 1. FIG. 1 illustrates an example ofthe device system according to the first exemplary embodiment.

The device system according to the first exemplary embodiment includes,for example, a terminal device 10, plural devices (for example, devices12 and 14), and a server 16 which is an example of an external device.In the example illustrated in FIG. 1, the terminal device 10, thedevices 12 and 14, and the server 16 have a function of communicatingwith each other via a communication path N such as a network. Theterminal device 10, the devices 12 and 14, and the server 16 maycommunicate with another device via different communication paths,rather than using the communication path N. In the example illustratedin FIG. 1, the device system includes two devices (the devices 12 and14). However, the device system may include three or more devices. Inaddition, the device system may include plural terminal devices 10 orplural servers 16. In addition, the device system may not include theserver 16.

The terminal device 10 is a device such as a personal computer (PC), atablet PC, a smart phone, or a mobile phone, and has a function oftransmitting and receiving data with another device. In addition, theterminal device 10 may be a wearable terminal (for example, a wristwatchtype terminal, a wristband type terminal, an eyeglass type terminal, aring type terminal, a contact lens type terminal, an intracorporealimplanting type terminal, or a hearable terminal). The terminal device10 functions as, for example, a user interface unit (UI unit) when thedevice is used.

Each of the devices 12 and 14 has a function and is, for example, animage forming device having an image forming function, a PC, a tabletPC, a smart phone, a mobile phone, a robot (for example, a humanoidrobot, an animal (other than human) type robot, or another type robot),a projector, a display device such as a liquid crystal display, arecording device, a playback device, an image capturing device such as acamera, a refrigerator, a rice cooker, a microwave oven, a coffee maker,a vacuum cleaner, a washing machine, an air conditioner, a lightingdevice, a watch, a surveillance camera, an automobile, a motorcycle, anaircraft (for example, an unmanned aircraft (so-called, drone)), a gamedevice, or various sensing devices (for example, a temperature sensor, ahumidity sensor, a voltage sensor, or a current sensor). Each of thedevices 12 and 14 may be a device that outputs an output to a user (forexample, an image forming device or a PC) or a device that outputs nooutput to the user (for example, a sensing device). In addition, all ofplural devices executing a linkage function which will be describedlater may output an output to the user. Some of the devices may outputan output to the user, and the other devices may not output an output tothe user. All the devices may not output an output to the user. Theconcept of the devices 12 and 14 may include all general kinds ofdevices. For example, the devices according to the present exemplaryembodiment may also include an information device, a video device, anaudio device, and other devices. In addition, the devices 12 and 14 havea function of transmitting and receiving data with other devices.

The server 16 is a device that manages, for example, data, userinformation, and the devices 12 and 14. Further, the server 16 has afunction of transmitting and receiving data with other devices.

In the terminal device 10, a conversation partner having a function ofmaking a conversation with a user is used. For example, the conversationpartner has a function of receiving a message of a user, analyze thecontent of the message, create a response such as a reply to themessage, and provide the response to the user. For example, the user'smessage is made by character input, voice input, image input or thelike. The response such as the reply is made by character output, voiceoutput, image output or the like. The conversation partner isimplemented by, for example, executing a program, and the program isinstalled in, for example, the terminal device 10. In addition, theprogram for the conversation partner may be installed in an externaldevice such as the server 16 such that the function of the conversationpartner may be provided from the external device such as the server 16to the terminal device 10.

The conversation partner described above is configured with, forexample, an automatic response artificial intelligence (AI) implementedby AI. The automatic response AI has a function of analyzing a messagecontent of a user and providing the user with a response such as a replyto the message content. The automatic response AI may be a so-calledchatbot (an automatic chatting program using AI). The automatic responseAI may have a learning function by AI and have an ability to make adetermination similar to a human decision through the learning function.In addition, neural network type deep learning may be used,reinforcement learning or the like for partially reinforcing a learningfield may be used, or a genetic algorithm, cluster analysis, aself-organization map, ensemble learning and others may be used. Inaddition, an AI related technology other than the technologies describedabove may be used.

When the automatic response AI is used, for example, a notification of afunction that each device has or a notification of a linkage functionexecutable by using plural functions is implemented while the user makesa conversation with the automatic response AI. In addition, a process ofspecifying a function that each device has or a linkage function may beperformed by the terminal device 10 or an external device such as theserver 16. Hereinafter, it is assumed that the specifying process isperformed by the terminal device 10.

Hereinafter, a configuration of the terminal device 10 will be describedin detail with reference to FIG. 2.

A communication unit 18 is a communication interface and has a functionof transmitting data to another device and a function of receiving datafrom another device. The communication unit 18 may be a communicationinterface having a wireless communication function or a communicationinterface having a wired communication function. For example, thecommunication unit 18 is compatible with one type or plural types ofcommunication methods and may communicate with a communication partneraccording to a communication method suitable for the communicationpartner (that is, a communication method with which the communicationpartner is compatible). The communication method is, for example, aninfrared communication, a visible light communication, a Wi-Fi(registered trademark) communication, or a near-field wirelesscommunication (for example, a near field communication (NFC)). As forthe near-field wireless communication, for example, Felica (registeredtrademark), Bluetooth (registered trademark), or a radio frequencyidentifier (RFID) is used. Another type of a wireless communication maybe used as the near-filed wireless communication. The communication unit18 may switch a communication method or a frequency band according to,for example, a communication partner or a surrounding environment.

A UI unit 20 is a user interface unit and includes a display unit and anoperation unit. The display unit is, for example, a display device suchas a liquid crystal display. The operation unit is, for example, aninput device such as a touch panel or a keyboard. In addition, the UIunit 20 may be a user interface serving as both the display unit and theoperation unit (for example, a touch type display or a device of whichdisplay electronically displays a keyboard or the like). In addition,the UI unit 20 may include a sound collecting unit such as a microphoneand a sound generating unit such as a speaker. In this case, informationmay be input to the terminal device 10 by a voice input or informationmay be output by voice.

The display unit of the UI unit 20 may display, for example, an imagecaptured by a camera, an image associated with a device identified as adevice to be used (for example, a device to be used alone or a device tobe linked), and an image associated with a function. The imageassociated with a device may be an image (a still image or a video)representing the device captured by a camera, or an image schematicallyrepresenting the device (for example, an icon). For example, the imagedata schematically representing the device may be created by acontroller 26 of the terminal device 10, stored in advance in a memory24 of the terminal device 10, stored in the server 16 to be providedfrom the server 16 to the terminal device 10, or stored in anotherdevice to be provided from the other device to the terminal device 10.The image associated with a function is, for example, an image such asan icon representing the function.

A camera 22 which is an image capturing unit generates image data (forexample, still image data or video data) by capturing an object to becaptured. The image is displayed on, for example, the display unit ofthe UI unit 20. The image displayed on the display unit may be operatedby the user. In addition, image data obtained by capturing with anexternal camera may be transmitted to the terminal device 10 via acommunication path, and the image may be displayed on the display unitof the UI unit 20. In this case as well, the image may be operated bythe user.

The memory 24 is a storage device such as a hard disk or a memory (forexample, an SSD). The memory 24 stores, for example, device functionmanagement information, linkage function management information, variousdata, various programs (for example, an operating system (OS), a program(software) for implementing the automatic response AI, and variousapplication programs (application software)), information indicating anaddress of each device (device address information), informationindicating an address of the server 16 (server address information),information on an identified device, information on an identified deviceto be linked, information on a function that an identified device has,and information on a linkage function. These pieces of information maybe stored in different storage devices or in a single storage device. Inaddition, the program for implementing the automatic response AI may bestored in an external device such as the server 16, such that theprogram itself may be executed by the external device, and the functionof the automatic response AI may be provided from the external device tothe terminal device 10. In this case, the program for implementing theautomatic response AI may not be stored in the memory 24.

Hereinafter, the device function management information and the linkagefunction management information will be described.

The device function management information is information for managingthe function that each device has, and represents, for example, anassociation between device identification information (deviceidentification information) for identifying a device and functioninformation indicating the function that the device has. The deviceidentification information is, for example, a device ID, a device name,information indicating a device type, a model number of a device,information for managing a device (for example, an asset managementnumber), information indicating a position where a device is installed(device position information), a device image associated with a device,and address information of a device. The device image is, for example,an external appearance image representing a device. The externalappearance image may be an image representing an outer side of a device(for example, a casing of a device), an image representing a state wherethe casing is opened such that the inside of the device is seen from theoutside (for example, the internal structure), or an image representinga state where the device is covered with, for example, a packing sheet.The device image may be an image generated by capturing a device (forexample, an image representing the outer side or the inside of a device)or an image emulating a device (for example, an icon). The functioninformation is, for example, a function ID or a function name. Forexample, in a case where the device 12 is an image forming device andthe image forming device has a printing function, a scanning function,and a copying function, the device identification information of theimage forming device is associated with, for example, functioninformation indicating the printing function, function informationindicating the scanning function, and function information indicatingthe copying function. The function that each device has is specified(identified) by referring to the device function management information.

A device managed by the device function management information is, forexample, a device included in the device system (for example, the device12 or 14). In addition, a device which is not included in the devicesystem may be managed by the device function management information. Forexample, the terminal device 10 may acquire information on a new devicewhich is not included in the device system (including the deviceidentification information and the function information), and newlyregister the information in the device function management information.The information on a device may be acquired by using, for example, theInternet or may be input by, for example, an administrator. In addition,the terminal device 10 may update the device function managementinformation, for example, at any timing, regularly, or at a timingdesignated by, for example, an administrator. Thus, function informationindicating the function that a device did not have before updating butthe device has after updating may be registered in the device functionmanagement information. Similarly, function information indicating thefunction that a device did not have before updating but the device hasafter updating may be deleted from the device function managementinformation or registered as unusable information. Information for theupdating may be acquired by using, for example, the Internet or may beinput by, for example, an administrator. In addition, the terminaldevice 10 itself may not acquire the information on a device to performthe generation or updating of the device function managementinformation. An external device such as the server 16 may acquire theinformation on a device and perform the generation or updating of thedevice function management information. In this case, the devicefunction management information generated or updated by the externaldevice is transmitted from the external device to the terminal device 10and stored in the memory 24.

The linkage function management information is information for managinga linkage function which is executed by linking plural functions to eachother. One or plural linkage functions is executed by linking pluralfunctions to each other. For example, the linkage function may be afunction executable by linking plural functions that a single device(for example, the device 12) has to each other, or a function executableby linking plural functions that plural devices (for example, thedevices 12 and 14) have to each other. In addition, the terminal device10 that makes an operation instruction may also be used as a device tobe linked, and the function that the terminal device 10 has may also beused as a portion of the linkage function.

The linkage function may be executed without using hardware devices. Forexample, the linkage function may be executed by linking plural piecesof software to each other. In addition, the linkage function may beexecuted by linking the function that a hardware device has and thefunction implemented by software to each other.

The linkage function management information represents, for example, anassociation between a combination of pieces of function informationindicating respective functions used for a linkage function and linkagefunction information indicating the linkage function. The linkagefunction information is, for example, a linkage function ID or a linkagefunction name. When a single function is updated, the linkage functionmanagement information is also updated according to the updating of thesingle function. Thus, a linkage function by plural functions which areunable to be linked to each other before updating may become usableafter updating, or in the meantime, a linkage function which is usablebefore updating may become unusable after updating. The linkage functioninformation indicating the linkage function which becomes usable afterupdating is registered in the linkage function management information,and the linkage function information indicating the linkage functionwhich becomes unusable after the updating is deleted from the linkagefunction management information or registered as unusable information.The updating of the linkage function management information may beperformed by the terminal device 10 or an external device such as theserver 16. When the updating of the linkage function managementinformation is executed by an external device, the updated linkagefunction management information is transmitted from the external deviceto the terminal device 10 and stored in the memory 24.

When plural devices are linked to each other, the linkage functionmanagement information is, for example, information for managing alinkage function using plural functions that the plural devices have,and indicates an association between a combination of pieces of deviceidentification information for identifying the respective devices usedfor the linkage function and the linkage function information. Inaddition, when a device having plural output functions (a device havingplural output functions to output an output to the user) and a sensingdevice which does not have the output function (a device outputting nooutput to the user) are linked to each other, the linkage functionmanagement information may include information indicating a combinationof the specific sensing device and a specific one of the plural outputfunctions. Accordingly, among the plural output functions, an outputfunction to be used in combination with the specific sensing device isspecified. As described above, when the device function managementinformation is updated, the linkage function management information isalso updated according to the updating of the device function managementinformation. Thus, a linkage function by plural devices which are unableto be linked to each other before updating may become usable afterupdating, or in the meantime, a linkage function which is usable beforeupdating may become unusable after updating.

The linkage function may be a function executable by linking pluraldifferent functions to each other or a function executable by linkingidentical functions to each other. The linkage function may be afunction which is unusable before the linkage. The function which isunusable before the linkage may be a function usable by using identicalfunctions among functions that devices to be linked have or a functionusable by combining different functions with each other among functionsthat devices to be linked have. For example, when a device having theprinting function (a printer) and a device having the scanning function(a scanner) are linked to each other, a copying function is implementedas a linkage function. That is, the copying function is implemented bylinking the printing function and the scanning function to each other.In this case, the copying function which is a linkage function isassociated with the combination of the printing function and thescanning function. For example, in the linkage function managementinformation, linkage function information indicating the copyingfunction as the linkage function is associated with the combination ofdevice identification information for identifying the device having theprinting function and device identification information for identifyingthe device having the scanning function.

The concept of the linkage function may include a coalescing functionwhich executes a new function by linking plural functions or pluraldevices to each other. For example, an extended displaying function as acoalescing function may be implemented by combining plural displays. Asanother example, a recording function as a coalescing function may beimplemented by combining a television and a recorder. The videorecording function may be a function of recording an image displayed onthe television. In addition, an image capturing area extension functionas a coalescing function may be implemented by combining plural cameras.The extension function is, for example, a function to capture an imageby linking the capturing areas of the respective cameras to each other.In addition, a translated call function (a function of translating aconversation via a telephone) as a coalescing function may beimplemented by combining a telephone with a translator or translationsoftware. As described above, the concept of the linkage functionincludes a function which may be implemented by linking identical typesof devices or functions to each other, and a function which may beimplemented by linking different types of devices or functions to eachother.

The controller 26 controls the operation of each unit of the terminaldevice 10. The controller 26 performs, for example, the control of acommunication by the communication unit 18 and a control of informationdisplay on the display unit of the UI unit 20. Further, the controller26 includes a specifying unit 28.

The specifying unit 28 has a function to specify a function that adevice designated by the user has or a linkage function executable byusing plural devices designated by the user. The specifying unit 28 isimplemented by, for example, the automatic response AI. In addition, thespecifying unit 28 may be implemented by another program or the likerather than the automatic response AI.

For example, the specifying unit 28 receives device identificationinformation for identifying a device designated by the user, andspecifies function information indicating a function associated with thedevice identification information in the device function managementinformation stored in the memory 24. Accordingly, the function that thedevice has is specified (identified).

The controller 26 controls a notification (guidance) of the informationon the function specified by the specifying unit 28 (for example,function information or function explanation information). As for thenotification, the controller 26 may display the information on thefunction on the display unit of the UI unit 20 or issue the informationon the function as voice information by using, for example, a speaker.In addition, the controller 26 may control execution of the function. Inaddition, the designation of a device by the user may be performed onthe screen of the display unit of the UI unit 20 or may be performed byvoice. For example, when an image associated with a device is displayedon the screen of the UI unit 20, the user may designate the device bydesignating the image on the screen. As another example, the user maydesignate a device by voice, regardless of whether an image associatedwith the device is displayed on the screen of the UI unit 20. Forexample, when the user utters a device name by voice, the voice iscollected by the sound collecting unit such as a microphone, and voiceinformation indicating the device name is input to the terminal device10. The specifying unit 28 determines that the device having the devicename is designated by the user, and specifies the function of thedevice. As described above, the operation to designate a device and thenotification of a function may be performed either by a display on thescreen or by voice. The display on the screen and the voice may becombined with each other. For example, the user may designate an imageassociated with a device on the screen so as to designate the device,and information indicating the function of the device may be displayedon the screen. As another example, a device may be designated in themanner that the user designates an image associated with the device onthe screen, and information indicating the function of the device may beoutput by voice. As further another example, a device may be designatedin the manner that a device name is input as voice information to theterminal device 10, and information indicating the function of thedevice may be displayed on the screen. As still further another example,a device may be designated in the manner that a device name is input asvoice information to the terminal device 10, and information indicatingthe function of the device may be output by voice.

In addition, the process of specifying a function may be performed bythe server 16. In this case, device identification information istransmitted from the terminal device 10 to the server 16, and functioninformation indicating a function associated with the deviceidentification information is specified by the specifying unit 28provided in the server 16. In addition, when the specifying process isperformed by the server 16, the server 16 stores the device functionmanagement information. The information on the function is transmittedfrom the server 16 to the terminal device 10, and may be displayed onthe display unit of the UI unit 20 or output as voice information.

In addition, the specifying unit 28 receives pieces of deviceidentification information for identifying respective devices to belinked, and specifies linkage function information indicating a linkagefunction associated with a combination of the pieces of the deviceidentification information in the linkage function managementinformation stored in the memory 24. Accordingly, the linkage functionexecutable by linking the functions that the respective devices to belinked have with each other is specified (identified). The controller 26controls a notification (guidance) of the information on the specifiedlinkage function (for example, linkage function information or linkagefunction explanation information). The controller 26 may display theinformation on the linkage function on the display unit of the UI unit20 or issues the information of the linkage function as voiceinformation. In addition, as described above, the designation of therespective devices to be linked may be performed on the screen or byvoice.

In addition, the process of specifying a linkage function may beperformed by the server 16. In this case, plural pieces of deviceidentification information are transmitted from the terminal device 10to the server 16, and linkage function information indicating a linkagefunction associated with the plural pieces of device identificationinformation is specified by the specifying unit 28 provided in theserver 16. In addition, when the specifying process is performed by theserver 16, the server 16 stores the linkage function managementinformation. The information on the linkage function is transmitted fromthe server 16 to the terminal device 10, and may be displayed on thedisplay unit of the UI unit 20 or output as voice information.

For example, when a single device is designated, a function that thedevice has is specified, and when plural devices are designated, alinkage function executable by using the plural devices is specified.

In addition, the specifying unit 28 may receive pieces of functioninformation indicating respective functions used for a linkage function,and specify linkage function information indicating a linkage functionassociated with a combination of the pieces of the function informationin the linkage function management information. Accordingly, the linkagefunction executable by linking the respective functions to be linked toeach other is specified (identified). The specifying process may also beperformed by the server 16, and the process result may be displayed onthe display unit of the UI unit 20 or output as voice information.

For example, a device is designated in the manner that the usertransmits information of the device to the automatic response AI. Whenthe user transmits information of plural devices to the automaticresponse AI, the specifying unit 28 identifies the plural devices asdevices to be linked and specifies a linkage function executable byusing the plural devices. In addition, when the user transmitsinformation of a single device to the automatic response AI, thespecifying unit 28 identifies the device and specifies a function thatthe device has. The information of a device is device identificationinformation for identifying the device and is, for example, a deviceimage associated with the device or character information or voiceinformation indicating the device identification information. Forexample, when the user transmits a device image associated with a deviceto the automatic response AI, when the user provides the automaticresponse AI with a character string indicating device identificationinformation of a device by inputting the character string to theterminal device 10, or when the user provides the automatic response AIwith device identification information of a device by voice, thespecifying unit 28 (the specifying unit 28 implemented by the automaticresponse AI) identifies the device.

In addition, a function usable by the user (a solo function that adevice has or a linkage function) may be managed for each user. Thismanagement may be performed by the terminal device 10 or an externaldevice such as the server 16. The function usable by the user is, forexample, a function provided to the user for free or a function providedto the user for a charge and purchased by the user. Usable functioninformation indicating a function usable by the user (for example,function purchase history information) may be created and managed foreach user. When the management of the usable function is performed bythe terminal device 10, the usable function information is stored in thememory 24, and when the management is performed by an external devicesuch as the server 16, the usable function information is stored in theexternal device. In addition, since the usable function also includes,for example, a function usable for free, an additional updatingfunction, and a function specifically managed by an administrator, theusable function may not be determined simply according to the purchaseof a function or not. A process of purchasing a function is performed byan external device such as the server 16.

The usable function management information is information for managing afunction usable by each user, and indicates, for example, an associationbetween user identification information for identifying a user andfunction information (which may include linkage function information)indicating a function usable by the user. As described above, thefunction usable by the user is, for example, a function provided to theuser for free or a function purchased by the user, and may be a singlefunction or a linkage function. The user identification information is,for example, user account information such as a user ID or a name of theuser. A function usable by each user is specified (identified) byreferring to the usable function management information. The usablefunction management information may be updated, for example, each time afunction is provided to the user (for example, each time a function isprovided to the user for free or for a charge).

When the function usable by the user is managed, the specifying unit 28may receive the user identification information for identifying the userand specify function information indicating each function associatedwith the user identification information in the usable functionmanagement information (for example, the information stored in theterminal device 10 or the server 16). Accordingly, the function usableby the user is specified (identified). The controller 26 may display theinformation on the specified function on the display unit of the UI unit20 or issue the information of the specified function as voiceinformation. For example, the specifying unit 28 receives deviceidentification information and user identification information. Then,the specifying unit 28 specifies function information indicating afunction associated with the device identification information in thedevice function management information, and specifies functioninformation indicating a function associated with the useridentification information in the usable function managementinformation. Accordingly, the function usable by the user that isspecified by the user identification information is specified as thefunction that the device specified by the device identificationinformation has.

In addition, the function usable by the user may be specified by anexternal device such as the server 16. In this case, the useridentification information is transmitted from the terminal device 10 tothe server 16, and function information indicating each functionassociated with the user identification information is specified by thespecifying unit 28 provided in the server 16. When the specifyingprocess is performed by the server 16, the server 16 stores the usablefunction management information. The information on each function usableby the user is transmitted from the server 16 to the terminal device 10,and may be displayed on the display unit of the UI unit 20 or output asvoice information.

Hereinafter, a configuration of the device 12 will be described indetail with reference to FIG. 3. For example, it is assumed that thedevice 12 is an image forming device. Hereinafter, the device 12 may bereferred to as an image forming device 12. FIG. 3 illustrates aconfiguration of the image forming device 12.

A communication unit 30 is a communication interface and has a functionof transmitting data to another device and a function of receiving datafrom another device. The communication unit 30 may be a communicationinterface having a wireless communication function or a communicationinterface having a wired communication function. For example, thecommunication unit 30 is compatible with one type or plural types ofcommunication methods and may communicate with a communication partneraccording to a communication method suitable for the communicationpartner (that is, a communication method with which the communicationpartner is compatible). The communication method is, for example, aninfrared communication, a visible light communication, a Wi-Ficommunication, or a near-field wireless communication. For example, thecommunication unit 30 may switch a communication method or a frequencyband according to a communication partner or a surrounding environment.

An image forming unit 32 has an image forming function. Specifically,the image forming unit 32 has at least one of a scanning function, aprinting function, a copying function, or a faxing function. When thescanning function is executed, a document is read, and scanned data(image data) is generated. When the printing function is executed, animage is printed on a recording medium such as a paper. When the copyingfunction is executed, a document is read and printed on a recordingmedium. When the faxing function is executed, image data is transmittedor received by facsimile. In addition, a linkage function using pluralfunctions may be executed. For example, a scanning transmitting functionwhich is a combination of a scanning function and a transmittingfunction may be executed. When the scanning transmitting function isexecuted, a document is read so that scanned data (image data) isgenerated, and transmitted to a transmission destination (for example,an external device such as the terminal device 10). In addition, thelinkage function is merely an example, and another linkage function maybe executed.

A memory 34 is a storage device such as a hard disk or a memory (forexample, an SSD). The memory 34 stores, for example, informationindicating an instruction of image formation (for example, jobinformation), image data to be printed, scanned data which is generatedby executing the scanning function, device address informationindicating an address of another device, information indicating anaddress of the terminal device 10 (terminal address information), serveraddress information of the server 16, various control data, and variousprograms. These pieces of information may be stored in different storagedevices or in a single storage device.

A UI unit 36 is a user interface unit including a display unit and anoperation unit. The display unit is, for example, a display device suchas a liquid crystal display. The operation unit is, for example, aninput device such as a touch panel or a keyboard. In addition, the UIunit 36 may be a user interface serving as both the display unit and theoperation unit (for example, a touch type display or a device of whichdisplay electronically displays a keyboard or the like). In addition,the image forming device 12 may not include the UI unit 36, and mayinclude a hardware user interface unit (a hardware UI unit) having nodisplay unit. The hardware UI unit is, for example, a hardware keyspecialized in input of numerals (for example, a numeric keypad) or ahardware key specialized in indication of directions (for example,direction indication keys).

A controller 38 controls the operation of each unit of the image formingdevice 12.

Hereinafter, a configuration of the server 16 will be described indetail with reference to FIG. 4. FIG. 4 illustrates a configuration ofthe server 16.

A communication unit 40 is a communication interface and has a functionof transmitting data to another device and a function of receiving datafrom another device. The communication unit 40 may be a communicationinterface having a wireless communication function or a communicationinterface having a wired communication function.

A memory 42 is a storage device such as a hard disk or a memory (forexample, an SSD). The memory 42 stores, for example, various data,various programs, address information of the terminal device 10, addressinformation of each device, and server address information of the server16. These pieces of information may be stored in different storagedevices or in a single storage device.

When the function that each device has and the linkage function aremanaged by the server 16, the above-described device function managementinformation and linkage function management information are stored inthe memory 42, such that the device function management information andthe linkage function management information may be updated in the server16. In this case, when the process of specifying a function is performedby the terminal device 10, the device function management informationand the linkage function management information (updated informationwhen information is updated) are transmitted from the server 16 to theterminal device 10. The transmission may be performed periodically or ata designated timing. Accordingly, the information stored in the terminaldevice 10 is updated. In addition, the specifying unit 28 of theterminal device 10 may specify the function that each device has orlinkage function by referring to the device function managementinformation and the linkage function management information that arestored in the server 16. In addition, the process of specifying thefunction may be executed by the server 16.

A controller 44 controls the operation of each unit of the server 16.The controller 44 controls, for example, the communication by thecommunication unit 40.

The controller 44 may execute the process of purchasing a function andmanage the history of the purchase. For example, when the user purchasesa paid function, the controller 44 may apply a charging process to theuser.

In addition, the controller 44 may execute the functions related toimage processing such as a character recognition function, a translationfunction, an image processing function, and an image forming function.In addition, the controller 44 may execute the function related to aprocess other than the image processing. When the character recognitionfunction is executed, characters in an image are recognized, andcharacter data indicating the characters is generated. When thetranslation function is executed, characters in an image are translatedinto characters expressed in a specific language, and character dataindicating the translated characters is generated. When the imageprocessing function is executed, an image is processed. For example, thecontroller 44 may receive scanned data that has been generated byexecuting the scanning function from the image forming device 12, andexecute the function related to the image processing such as thecharacter recognition function, the translation function, or the imageprocessing function, for the scanned data. The controller 44 may receiveimage data from the terminal device 10 and execute each function for theimage data. For example, the character data or the image data generatedby the controller 44 is transmitted from the server 16 to the terminaldevice 10. The server 16 may be used as an external device, and alinkage function may use functions that plural devices including theserver 16 have.

As described above, the device function management information may bestored in the memory 42 of the server 16. In this case, the devicefunction management information may not be stored in the memory 24 ofthe terminal device 10. Similarly, the linkage function managementinformation may be stored in the memory 42 of the server 16. In thiscase, the linkage function management information may not be stored inthe memory 24 of the terminal device 10. The controller 44 of the server16 includes the specifying unit 28 described above, and may specify thefunction that a device has or a linkage function executable by usingplural functions, by identifying a device based on the deviceidentification information. In this case, the terminal device 10 may notinclude the specifying unit 28.

When the usable function management information is created, the usablefunction management information may be stored in the memory 42 of theserver 16. In this case, the usable function management information maynot be stored in the memory 24 of the terminal device 10. The controller26 of the terminal device 10 may manage the history of purchase of afunction by the user. In this case, the controller 44 of the server 16may not have the function of managing the history.

As another example, a device such as the device 12 or 14 may store thedevice function management information or the linkage functionmanagement information, or the device such as the device 12 or 14 mayinclude the specifying unit 28. That is, the process by the specifyingunit 28 may be performed by the terminal device 10, a device such as thedevice 12 or 14, the server 16, or another device.

In the first exemplary embodiment, the device identification informationis acquired by applying, for example, the augmented reality (AR)technique so that a device is identified. For example, by applying theAR technique, device identification information of a device used aloneis acquired so that the device is identified, and device identificationinformation of devices to be linked is acquired so that the devices tobe linked are identified. A known AR technique may be used as the ARtechnique. For example, a marker AR technique using a marker such as atwo-dimensional barcode, a markerless AR technique using an imagerecognition technique, or a position information AR technique usingposition information is used. In addition, the device identificationinformation may be acquired without using the AR technique so that adevice is identified. For example, when a device is connected to anetwork, the device may be identified based on an IP address, or deviceID may be read so that the device may be identified. In addition, when adevice or a terminal device having various wireless communicationfunctions such as an infrared communication, a visible lightcommunication, Wi-Fi, and Bluetooth is used, device ID may be acquiredby using the wireless communication functions so that the device isidentified.

Hereinafter, the process of acquiring device identification informationwill be described in detail with reference to FIG. 5. As an example, acase where device identification information of the image forming device12 is acquired will be described. FIG. 5 schematically illustrates theexternal appearance of the image forming device 12. Here, the process ofacquiring device identification information by applying the marker ARtechnique will be described. A marker 46 such as a two-dimensionalbarcode is provided on a casing of the image forming device 12. Themarker 46 is information obtained by encoding device identificationinformation of the image forming device 12. The user activates thecamera 22 of the terminal device 10 and captures the marker 46 providedon the image forming device 12 to be used, using the camera 22.Accordingly, image data representing the marker 46 is generated. Thecontroller 26 of the terminal device 10 extracts the deviceidentification information by applying a decoding process to the markerimage represented in the image data. Accordingly, the image formingdevice 12 to be used (the image forming device 12 having the capturedmarker 46) is identified. The specifying unit 28 of the terminal device10 specifies function information indicating a function associated withthe extracted device identification information, in the device functionmanagement information. Accordingly, the function that the image formingdevice 12 to be used has is specified (identified).

In addition, the controller 44 of the server 16 may extract the deviceidentification information by applying the decoding process to the imagedata representing the marker 46. In this case, the image data istransmitted from the terminal device 10 to the server 16, and thedecoding process is applied to the image data in the server 16. When theserver 16 is provided with the specifying unit 28, the functionassociated with the device identification information may be specifiedin the server 16.

The function information indicating the function that the image formingdevice 12 has may be encoded and included in the marker 46. In thiscase, when the decoding process is applied to the image datarepresenting the marker 46, the device identification information of theimage forming device 12 is extracted, and the function informationindicating the function that the image forming device 12 has is alsoextracted. Accordingly, the image forming device 12 is specified(identified), and the function that the image forming device 12 has isspecified (identified). The decoding process may be performed by theterminal device 10 or the server 16.

In addition, device identification information of each of plural devicesis acquired by capturing each marker in the plural devices.

For example, when device identification information is acquired byapplying the markerless AR technique, the user captures the entireexternal appearance or a portion of the external appearance of a deviceto be used (for example, the image forming device 12), with the camera22 of the terminal device 10. In order to acquire information forspecifying a device to be used such as a name (for example, a productname), a model number, or an asset management number, it is helpful tocapture the external appearance of the device. By the capturing,external appearance image data representing the entire externalappearance or a portion of the external appearance of the device to beused is generated. The controller 26 of the terminal device 10identifies the device to be used based on the external appearance imagedata. For example, the memory 24 of the terminal device 10 storesexternal appearance image association information, for each device,indicating an association between external appearance image datarepresenting the entire external appearance or a portion of the externalappearance of the device and device identification information of thedevice. For example, the controller 26 compares the external appearanceimage data with each external appearance image data included in theexternal appearance image association information, and specifies thedevice identification information of the device to be used based on thecomparison result. For example, the controller 26 extracts an externalappearance feature of the device to be used from the acquired externalappearance image data, specifies external appearance image datarepresenting the same or similar external appearance feature as or tothe extracted external appearance feature, in the external appearanceimage data group included in the external appearance image associationinformation, and specifies device identification information associatedwith the external appearance image data. Accordingly, the device to beused (the device captured by the camera 22) is identified. As anotherexample, when external appearance image data representing a name (forexample, a product name) or a model number of the device to be used isgenerated by capturing the name or the model name, the device to be usedmay be identified based on the name or the model number represented inthe external appearance image data. The specifying unit 28 of theterminal device 10 specifies function information indicating eachfunction associated with the specified device identificationinformation, in the device function management information. Accordingly,the function that the device to be used (for example, the image formingdevice 12) has is specified.

In addition, the controller 44 of the server 16 may compare the externalappearance image data representing the entire external appearance or aportion of the external appearance of the device to be used (forexample, the image forming device 12) with each appearance image dataincluded in the external appearance image association information, andspecify the device identification information of the device to be usedbased on the comparison result. The external appearance imageassociation information is stored in the memory 42 of the server 16. Inthis case, the controller 44 of the server 16 specifies the deviceidentification information of the device to be used, by referring to theexternal appearance image association information.

Device identification information of each of plural devices is acquiredby capturing the entire external appearance or a portion of the externalappearance of each device.

When device identification information is acquired by applying theposition information AR technique, position information indicating aposition where a device (for example, the image forming device 12) isinstalled is acquired by using, for example, a global positioning system(GPS) function. For example, each device has the GPS function andacquires device position information indicating a position of the owndevice. The terminal device 10 outputs information indicating a requestfor acquiring device position information to a device to be used, andreceives the device position information of the device from the deviceas a response to the acquisition request. The controller 26 of theterminal device 10 identifies the device to be used based on the deviceposition information. For example, the memory 24 of the terminal device10 stores position association information, for each device, indicatingan association between device position information indicating a positionwhere a device is provided and device identification information of thedevice. The controller 26 specifies device identification informationassociated with the device position information in the positionassociation information. Accordingly, the device to be used is specified(identified). The specifying unit 28 of the terminal device 10 specifiesfunction information indicating each function associated with thespecified device identification information, in the device functionmanagement information. As a result, the function that the device to beused (for example, the image forming device 12) has is specified(identified).

In addition, the controller 44 of the server 16 may specify the deviceidentification information associated with the position information ofthe device to be used in the position association information. Theposition association information is stored in the memory 42 of theserver 16. In this case, the controller 44 of the server 16 specifiesthe device identification information of the device to be used, byreferring to the position association information.

Device position information of each of plural devices is acquired, anddevice identification information of each device is specified based onthe device position information.

In addition, a device may be identified by using plural identificationtechniques. For example, a device may be identified by using pluraltechniques selected from the marker AR technique, the markerless ARtechnique, and the position information AR technique. In addition, whena device is not identified by a certain identification technique, thedevice may be identified by using another identification technique. Forexample, when a device is not identified by the marker AR technique orthe markerless AR technique, the device may be identified by using theposition information AR technique.

Hereinafter, the device system according to the first exemplaryembodiment will be described in more detail.

The device function management information will be described in detailwith reference to FIG. 6. FIG. 6 illustrates an example of a devicefunction management table which is the device function managementinformation. In the device function management table, for example, adevice ID, information indicating a device name (for example, a devicetype), information indicating a function that a device has (functioninformation), and an image ID are associated with each other. The deviceID and the device name correspond to an example of the deviceidentification information. The image ID is an example of imageidentification information for identifying a device image associatedwith a device. In addition, the device function management table may notinclude the image ID. For example, a device with the device ID “B” is amultifunction machine (an image forming device having plural imageforming functions), and has functions such as the printing function andthe scanning function. The device is associated with an image ID foridentifying a device image associated with the device. For example, dataof a device image associated with a device may be stored in the terminaldevice 10, the server 16, or another device.

For example, a device ID for identifying a device to be used is acquiredby applying the AR technique. The specifying unit 28 of the terminaldevice 10 specifies the device name, the function, and the image IDwhich are associated with the device ID, by referring to the devicefunction management table. Accordingly, the device to be used isidentified. The information indicating the device name or the deviceimage may be displayed on the display unit of the UI unit 20. Inaddition, an image itself captured by the camera 22 may be displayed onthe display unit of the UI unit 20. In addition, when a device imageassociated with the device (for example, an image captured by the camera22 or an image schematically representing the device) is designated bythe user in the UI unit 20, information on a function that the devicehas (for example, function information or function explanationinformation) may be displayed on the display unit of the UI unit 20.

Hereinafter, the linkage function management information will bedescribed in detail with reference to FIG. 7. FIG. 7 illustrates anexample of a linkage function management table which is the linkagefunction management information. In the linkage function managementtable, for example, a combination of device IDs, information indicatinga device name of each device to be linked (for example, types ofrespective devices), and information indicating a linkage function(linkage function information) are associated with each other. Forexample, a device with the device ID “A” is a personal computer (PC),and a device with the device ID “B” is a multifunction machine. When thePC A and the multifunction machine B are linked to each other, forexample, the “scanning transmitting function” or the “printing function”is implemented as a linkage function. The “scanning transmittingfunction” is a function of transmitting image data generated by scanningby the multifunction machine B to the PC A. The “printing function” is afunction of transmitting data (for example, image data or document data)saved in the PC A to the multifunction machine B and print the data bythe multifunction machine B. In addition, a linkage function executableby using three or more devices may be set. In this case, a combinationof the three or more devices and a linkage function are associated witheach other, and the association is registered in the linkage functionmanagement table.

Hereinafter, the operation of the terminal device 10 will be describedin detail by using specific examples.

FIG. 8 illustrates an example of a screen which is displayed on theterminal device 10. The controller 26 of the terminal device 10 displaysa screen 48 on the display unit of the UI unit 20 to and displaysvarious pieces of information of the screen 48. The screen 48 is aninterface (an interaction response screen) on which the user makes aconversation with the automatic response AI. The screen 48 displaysinformation (for example, a character string or an image) input by theuser to the terminal device 10, and information (for example, acharacter string or an image) indicating a message content of theautomatic response AI. The conversation between the user and theautomatic response AI may be a so-called chatting type conversation(that is, a real-time conversation. In further other words, aconversation conducted in the manner that the user and the automaticresponse AI post messages to each other). For example, when a programfor making a conversation with the automatic response AI (for example, achatting program) is executed, the screen 48 for the conversation (theinteraction response screen) is displayed on the display unit of the UIunit 20.

A display area for each party making a conversation may be formed on thescreen 48, and a message content of each party making a conversation maybe displayed in the own display area. For example, a display area 48 ais an area that displays a message content of the automatic response AI,and a display area 48 b displays a message content of the user(information input by the user to the terminal device 10). In theexample illustrated in FIG. 8, the display area 48 a for the automaticresponse AI is provided at the left side of the screen 48 in thedirection toward the screen 48, and the display area 48 b for the useris provided at the right side of the screen 48. In addition, the exampleof the display is merely an example. The arrangement relationship of thedisplay areas may be reversed, the respective display areas may beprovided vertically, the respective display areas may entirely orpartially overlap with each other, or a specific area of the screen 48(for example, the center area) may display the latest message content,and the periphery of the specific area may display past messagecontents. In addition, the respective display areas may have the samesize or different sizes from each other, or the size of each displayarea may be expanded or reduced according to an amount of messagecontents for each message or the number of times a messages is made. Thecontroller 26 performs the control of the setting or size of eachdisplay area.

In the example illustrated in FIG. 8, the parties making a conversationare one user and one automatic response AI. However, plural users orplural automatic response AIs may join the conversation. In this case, adisplay area for each of the parties making a conversation may beprovided on the screen 48. For example, when two users and one automaticresponse AI join the conversation, three display areas are provided onthe screen 48, and a message content of each party making a conversationis displayed in the own display area. For example, when plural usersconduct a conversation with each other via the screen 48 by conducting acommunication among plural terminal devices 10, message contents of theplural users are displayed on the screen 48. This display is identicallyapplied to the case where plural automatic response AIs join theconversation. Programs for plural automatic response Ms may be installedin the same terminal device 10 such that the plural automatic responseMs are stored in the same terminal device 10, or programs for pluralautomatic response AIs may be installed in different terminal devices 10such that the automatic response AIs are stored in the differentterminal devices 10, respectively. For example, the screen 48 displays amessage content of the automatic response AI stored in the own terminaldevice 10, and further, message contents of the automatic response AIsstored in other terminal devices 10. The automatic response AIs storedin the other terminal devices 10 join the conversation via, for example,the communication path N.

For example, when a chatting program is activated, the controller 26 ofthe terminal device 10 displays the screen 48 on the display unit of theUI unit 20. An image 50 associated with the automatic response AI isdisplayed in the display area 48 a for the automatic response AI, and animage 52 associated with the user is displayed in the display area 48 bfor the user. The image 50 identifies the automatic response AI, and theimage 52 identifies the user. In addition, a character string indicatingthe automatic response AI (a character string for identifying theautomatic response AI) may be displayed in place of or along with theimage 50. Similarly, a character string indicating the user (forexample, a character string for identifying the user such as a name, auser ID, a handle name, or a nickname) may be displayed in place of oralong with the image 52.

Then, a conversation is conducted between the user and the automaticresponse AI. In the conversation, the message content of the user may beinput to the terminal device 10 in the manner that the user inputsinformation such as a character string or an image by operating the UIunit 20, or may be input by voice. In addition, the message content ofthe user may or may not be displayed as information such as a characterstring or an image on the screen 48. Similarly, the message content ofthe automatic response AI may be displayed as information such as acharacter string or an image on the screen 48, or may be or may not bedisplayed on the screen 48 along with output by voice. Hereinafter, forexample, descriptions will be made by assuming that a message content ofeach party making a conversation is displayed on the screen 48. However,the message content may be output by voice.

In addition, when a conversation is conducted on the screen 48, thescreen 48 is scrolled so that past message contents which may not bedisplayed on the screen 48 disappear from the screen 48, and the latestmessage content and nearby message contents thereof are displayed on thescreen 48.

When the chatting program is activated, first, the automatic response AImakes a common question (a question asking about a user's request) suchas “How can I help you?,” and a character string indicating the questionis displayed as a message content of the automatic response AI on thescreen 48 (not illustrated in FIG. 8). In addition, the message contentof the automatic response AI may be output by voice in place of or alongwith the character string. Since the message content is one made by theautomatic response AI, the controller 26 display the message content inthe display area 48 a as a message content associated with the image 50of the automatic response AI. The same is applied to subsequentconversations.

Next, when the user requests for a linkage function by putting amessage, for example, “I want to execute a linkage function,” acharacter string indicating the request is displayed as a messagecontent of the user on the screen 48 (not illustrated in FIG. 8). Forexample, the user may input information indicating the message contentto the terminal device 10 by inputting a character string by operatingthe screen 48, or may input the information indicating the messagecontent to the terminal device 10 by voice. Since the message content ismade by the user, the controller 26 displays the message content in thedisplay area 48 b as a message content associated with the image 52 ofthe user. The same is applied to subsequent conversations.

The automatic response AI identifies the request of the user byanalyzing the message content of the user, and creates a response suchas a reply to the request. The controller 26 displays a character stringindicating the response in the display area 48 a as a message content 54of the automatic response AI. For example, the message content 54 that“Which device do you want to use?” is displayed in the display area 48 aas a message content of the automatic response AI. In addition, theautomatic response AI may search for information by using, for example,the Internet and analyze the search result, or may acquire informationon the devices 12 and 14 and analyze the information. The automaticresponse AI may reflect the analysis result on the response. Inaddition, the automatic response AI may manage, for example, the historyof use of various applications by the user, the history of an operationof the terminal device 10 by the user, and the history of use of thedevices 12 and 14 by the user. In addition, the automatic response AImay acquire information from the automatic response AIs stored in theother different devices by communicating with the automatic responseAIs. The automatic response AI may reflect the information managed bythe own automatic response AI or the acquired information, on theresponse.

When the user designates devices in response to the question above, thespecifying unit 28 (the specifying unit 28 implemented by the automaticresponse AI) identifies the devices as devices to be linked. Forexample, as illustrated in FIG. 8, when the user designates a deviceimage 56 associated with a device and places the device image 56 at theposition where a message content of the user is displayed, thecontroller 26 transmits the device image 56 as a message content of theuser to the automatic response AI. In response to this operation, thespecifying unit 28 identifies the device associated with the deviceimage 56 as a device to be linked. In the example illustrated in FIG. 8,the operation by the user to display the device image 56 in the displayarea 48 b corresponds to an act for the user to make a message content.In addition, the user may input a character string indicating a deviceas a message content of the user to the terminal device 10 by operatingthe UI unit 20. In this case, the specifying unit 28 identifies thedevice indicated by the character string as a device to be linked. Inaddition, information of a device to be linked may be input by voice asa message content of the user to the terminal device 10. In addition,the user may designate a type of a device, rather than designating anindividual device. In this case, the type of the device is identified asa type of a device to be linked. For example, the user may designate adevice itself such as the multifunction machine B or may designate adevice type such as a multifunction machine.

The device image 56 is an image associated with the multifunctionmachine B (the image forming device) serving as a device. Themultifunction machine B is a device identified by the specifying unit 28by using the AR technique described above (the marker AR technique, themarkerless AR technique, or the position information AR technique) oranother identification technique. In addition, the device image 56 maybe an image associated with a device type such as a multifunctionmachine, rather than the multifunction machine B itself.

For example, the controller 26 displays a stock area 58 in the displayarea 48 b for the user, and displays device images in the stock area 58.The stock area 58 is an area where information of one or pluralcandidates for objects to be linked is displayed, and in the exampleillustrated in FIG. 8, device images are displayed as an example of theinformation of the candidates. For example, the controller 26 displaysdevice images associated with already identified devices in the stockarea 58. In the example illustrated in FIG. 8, device images 56, 60, and62 are displayed in the stock area 58. The device image 60 is an imageassociated with the PC A which is a device. The device image 62 is animage associated with a projector C which is a device. As describedabove, for the identified devices, the controller 26 displays deviceimages associated with the devices in the stock area 58. In addition,the controller 26 may display device images associated with devicetypes, rather than devices themselves, in the stock area 58. Forexample, the controller 26 may display, in the stock area 58, the deviceimage 56 associated with a device type such as a multifunction machinerather than the multifunction machine B itself, the device image 60associated with a device type such as a PC rather than the PC A itself,and the device image 62 associated with a device type such as aprojector rather than the projector C itself.

For example, the information on the already identified devices is storedin the memory 24, and the device images associated with the devices aredisplayed in the stock area 58 later even when the process ofidentifying the devices is not performed again. In addition, theinformation on the identified devices may be deleted from the memory 24through an operation by the user. In this case, the device imagesassociated with the devices are also deleted. As another example, theinformation on the identified devices may be deleted from the memory 24after a linkage function is executed by using the identified devices orafter a predetermined time elapses from the time when the devices areidentified.

The device image 56 may be an image generated by capturing themultifunction machine B by the camera 22 (an image having a size at thecapturing time or an enlarged or reduced image), or a schematic image(for example, an icon) associated with the multifunction machine B.Similarly, the device image 60 may also be an image generated bycapturing the PC A or a schematic image associated with the PC A.Similarly, the device image 62 may also be an image generated bycapturing the projector C or a schematic image associated with theprojector C.

When a schematic image is used, the specifying unit 28 specifies theschematic image associated with the identified device by referring tothe device function management table illustrated in FIG. 6. Theschematic image is displayed as a device image. The data of theschematic image may be stored in the terminal device 10 or an externaldevice such as the server 16.

In addition, when image data generated by capturing a device is used,the image reflects the current external appearance of the device itself(for example, the external appearance reflecting a scratch, a memo, anda seal adhered to a device), and thus, there is an effect in that theuser may visually confirm the difference from a similar type of otherdevices. Further, since a device image associated with a device itself(for example, an image obtained by capturing or an icon associated witha device itself) is a unique image of the device, the automatic responseAI may identify the device itself as a device to be linked when thedevice image is transmitted to the automatic response AI. Meanwhile,when the device image is an image associated with a device type ratherthan a device itself, the automatic response AI may identify the devicetype from the device image, but may not identify an individual deviceitself. For example, when the device image 56 is an image associatedwith the multifunction machine B, the automatic response AI may identifythe multifunction machine B itself as a device to be linked, from thedevice image 56. Meanwhile, when the device image 56 is an imageassociated with a device type such as a multifunction machine, theautomatic response AI identifies the device type such as a multifunctionmachine as a type of a device to be linked, from the device image 56. Inthe descriptions hereinbelow, it is assumed that each device image is animage associated with an individual device itself. That is, it isassumed that the device image 56 is an image associated with themultifunction machine B itself, the device image 60 is an imageassociated with the PC A itself, and the device image 62 is an imageassociated with the projector C itself.

In addition, when a device is identified, information indicating thename of the device may be displayed on the screen 48. For example, acharacter string “Multifunction Machine B” may be displayed along withthe device image 56.

In addition, when there exists another device image that may not bedisplayed in the stock area 58, the other device image is displayed inthe stock area 58 in the manner that the user performs an operation toscroll the device images displayed in the stock area 58, as indicated byan arrow 64.

As indicated by an arrow 66, the user may select an image associatedwith a device to be linked (the device image 56 in the exampleillustrated in FIG. 8) from the stock area 58, move the image to theoutside of the stock area 58, and transmit the device image 56 as amessage content of the user to the automatic response AI. For example,the user moves the device image 56 from the stock area 58 to theposition where a message content of the user is displayed, by performinga dragging operation, and performs a dropping operation at the positionwhere a message content is displayed. With the operations, thecontroller 26 transmits the device image 56 as a message content of theuser to the automatic response AI.

In the example illustrated in FIG. 8, in addition to the device image56, the user selects the device image 60 from the stock area 58 andmoves the device image 60 to the position where a message content of theuser is displayed. With the operation, the device image 60 istransmitted as a message content of the user to the automatic responseAI.

When the user transmits information of the plural devices to theautomatic response AI on the screen 48, that is, when the user makes amessage indicating information of the plural devices as information ofdevices to be linked, the specifying unit 28 specifies linkage functionsassociated with the plural devices in the linkage function managementtable illustrated in FIG. 7. The controller 26 displays the informationon the linkage functions in the display area 48 a as a message contentof the automatic response AI.

In the example described above, the device images 56 and 60 aretransmitted as information of devices to be linked to the automaticresponse AI. In this case, the specifying unit 28 identifies themultifunction machine B associated with the device image 56 and the PC Aassociated with the device image 60 as devices to be linked. Inaddition, the device image 56 may be first transmitted to the automaticresponse AI, and then, the device image 60 may be transmitted to theautomatic response AI, or the device image 60 may be first transmittedto the automatic response AI, and then, the device image 56 may betransmitted to the automatic response AI.

For example, when the user selects and extracts images (the deviceimages 56 and 60 in the example described above) from the stock area 58by using a pointer (for example, the user's finger, a pen, or a stylus)on the screen 48, the images are included in a message content of theuser to be transmitted to the automatic response AI. For example, thedevice images are moved from the stock area 58 to the position where amessage content of the user is displayed, by the dragging and droppingoperations, and transmitted as a message content of the user to theautomatic response AI. The controller 26 detects the movement of thepointer on the screen 48 by detecting the contact of the pointer withthe screen 48.

When the device images 56 and 60 are transmitted to the automaticresponse AI, and the multifunction machine B and the PC A are identifiedas devices to be linked as described above, the controller 26 displaysthe information of the devices identified in the display area 48 a asdevices to be linked, as a message content 68 of the automatic responseAI. When the user approves the message content 68 (for example, when theuser makes a message content 70 answering “Yes”), information on linkagefunctions executable by using the PC A and the multifunction machine Bis displayed on the screen 48.

The specifying unit 28 specifies the linkage functions associated withthe combination of the PC A and the multifunction machine B (forexample, the “scanning transmitting function” and the “printingfunction”) in the linkage function management table illustrated in FIG.7, at a timing when the device images 56 and 60 are transmitted asinformation of devices to be linked to the automatic response AI, or ata timing when the user approves the message content 68. Accordingly, thelinkage functions executable by using the PC A and the multifunctionmachine B are specified. The “scanning transmitting function” is afunction of transmitting image data generated by scanning by themultifunction machine B, to the PC A. The “printing function” is afunction of transmitting image data stored in the PC A to themultifunction machine B and print the image data by the multifunctionmachine B. The information on the linkage functions specified asdescribed above is displayed as a message content of the automaticresponse AI on the screen 48. In addition, the process of specifying thedevices or the linkage functions may be performed by the server 16.

FIG. 9 illustrates the screen 48 which is continuous to FIG. 8. When thelinkage functions are specified as described above, the controller 26 ofthe terminal device 10 displays the information on the linkage functionsin the display area 48 a as a message content 72 of the automaticresponse AI.

When the user designates a linkage function and gives an instruction toexecute the linkage function by a message content 74 in response to themessage content 72, the controller 26 (for example, the controller 26implemented by the automatic response AI) receives the executioninstruction and transmits information indicating the executioninstruction (execution instruction information) to the devices to belinked. Further, the controller 26 displays information indicating thatthe execution instruction has been received, in the display area 48 a asa message content 76 of the automatic response AI. As another example,the screen 48 may display a button image for instructing execution of alinkage function, and the user may press the button image to make theinstruction to execute the linkage function. When the PC A and themultifunction machine B are selected as devices to be linked, theexecution instruction information is transmitted to the PC A and themultifunction machine B. Upon receiving the execution instructioninformation, the PC A and the multifunction machine B execute thelinkage function indicated in the execution instruction information (thelinkage function designated by the user). For example, when aninstruction to execute the “printing function” as a linkage function ismade, the PC A transmits image data to be printed to the multifunctionmachine B, and the multifunction machine B prints the image data on apaper which is a recording medium.

In addition, when the user designates the linkage function by themessage content 74, the controller 26 (the automatic response AI) maycause the UI unit 20 to display a screen for controlling the devices tobe linked, without giving the instruction to execute the linkagefunction to the devices to be linked. FIG. 10 illustrates a screen 78for that case. The screen 78 is, for example, a screen for controllingthe PC A and the multifunction machine B (a remote control screen) bythe terminal device 10. The screen 78 displays, for example, buttonimages for instructing execution of linkage functions (for example, abutton image for instructing the execution of the “scanning transmittingfunction” and a button image for instructing the execution of the“printing function”). When the user presses a button image on the screen78, the linkage function associated with the button image is executed.In addition, for example, setting of various parameters may be performedon the screen 78.

FIG. 11 illustrates another remote control screen. The screen 78includes a screen 78 a for controlling the multifunction machine B and ascreen 78 b for controlling the PC A. The screen 78 a may display, forexample, a button image for instructing execution of a solo functionthat the multifunction machine B has (for example, a button image forinstructing execution of the “copying function” or a button image forinstructing execution of the “printing function”). In addition, forexample, setting of various parameters used in the multifunction machineB may be performed on the screen 78 a. The screen 78 b displays, forexample, data (file) stored in the PC A or a storage location (folder)in the PC A. In addition, for example, setting of various parametersused in the PC A may be performed on the screen 78 b. In addition,setting of a linkage function (for example, the “scanning transmittingfunction” or the “printing function”) or an instruction to execute thelinkage function may be made on the screens 78 a and 78 b.

In addition, the device images displayed in the stock area 58 in FIG. 8or the information on the devices associated with the device images maybe edited by the user. For example, when the user designates a deviceimage displayed in the stock area 58 and gives an instruction to editthe device image, the controller 26 displays an editing screen on thedisplay unit of the UI unit 20. The controller 26 displays the editingscreen on the display unit of the UI unit 20, for example, when the userpresses the device image for a long time (for example, the length oftime during which the device image is pressed becomes equal to or longerthan a threshold), when the user presses a button image for displayingthe editing screen, or when the user inputs an editing instruction tothe terminal device 10 by voice. On the editing screen, the device imagemay be edited, and for example, a portion image associated with aportion of the device may be cut out from the device image. In addition,the information on the device associated with the device image may alsobe edited.

In addition, the controller 26 may display a device image associatedwith a device established for a communication with the terminal device10 in the stock area 58, or a device image associated with a deviceregistered in the terminal device 10 in the stock area 58.

In addition, when respective devices are identified by the marker ARtechnique or the markerless AR technique, the devices may beindividually captured and identified, or may be captured all togetherand identified. For example, the multifunction machine B is captured ina state where the multifunction machine B is included in the capturingarea of the camera 22, and then, the PC A is captured in a state wherethe PC A is included in the capturing area of the camera 22, so that themultifunction machine B and the PC A are sequentially identified. Therespective devices to be linked may not be arranged near to each other,and thus, the multifunction machine B and the PC A may not be includedtogether in the capturing area of the camera 22. This circumstance maybe coped with by, for example, changing the angle of the capturing area,or enlarging or reducing the capturing area. However, this operation maynot cope with the circumstance. In this case, the respective devices areidentified by capturing the devices plural times.

As another example, a device to be linked may be preset as a basiclinkage device. For example, it is assumed that the multifunctionmachine B is preset as a basic linkage device. The device identificationinformation of the basic linkage device may be stored in advance in thememory 24 of the terminal device 10. The user may designate the basiclinkage device by using the UI unit 20. When the basic linkage device isset, the user transmits a device image associated with a device to belinked other than the basic linkage device as a message content to theautomatic response AI. Accordingly, the device to be linked is specified(identified), and a linkage function executable by using the basiclinkage device and the specified device is specified (identified).

Hereinafter, operations when a linkage function is executed will bedescribed. For example, when a linkage function is executed, an accessrequest is transmitted from the terminal device 10 to each of thedevices to be linked, and the terminal device 10 is connected to each ofthe devices. Hereinafter, the access process will be described withreference to FIG. 12. FIG. 12 is a sequence diagram illustrating theprocess.

In the terminal device 10, when the user designates the multifunctionmachine B and the PC A as devices to be linked (that is, when the usertransmits the information of the multifunction machine B and the PC A asa conversation content of the user to the automatic response AI), theterminal device 10 transmits information indicating an access request tothe devices to be linked (the multifunction machine B and the PC A)(S01). The control of the transmission may be performed by the automaticresponse AI. For example, when address information of each of thedevices to be linked is stored in the server 16, the terminal device 10acquires the address information of each of the devices from the server16. When the device identification information of each of the devicesincludes the address information, the controller 26 of the terminaldevice 10 acquires the address information from the deviceidentification information. The process of acquiring the addressinformation may be performed by the automatic response AI. In addition,the address information of each of the devices may be stored in thememory 24 of the terminal device 10. The terminal device 10 may acquirethe address information of each of the devices by another method. Theterminal device 10 transmits the information indicating the accessrequest to each of the devices to be linked (for example, themultifunction machine B and the PC A), by using the address informationof each of the devices to be linked.

Upon receiving the information indicating the access request, themultifunction machine B and the PC A determine whether or not to permitthe access to the terminal device 10 (S02). For example, when themultifunction machine B and the PC A correspond to devices to whichaccess is not permitted, or when the number of devices requesting anaccess exceeds an upper limit, the access is not permitted. In addition,when the access from the terminal device 10 is permitted, the terminaldevice 10 may be prohibited from performing an operation to change theunique setting information of each of the multifunction machine B andthe PC A. For example, it may be prohibited to change color parametersof the multifunction machine B or setting time for shifting to powersaving. Thus, the security of the devices to be linked is improved. Asanother example, when the devices are linked to each other, the changeof the setting information may be restrictively performed, as comparedwith a case where each device is used alone. For example, it may bepermitted to change a smaller number of setting items, as compared witha case where a device is used alone. In addition, it may be prohibitedto refer to personal information of another user such as the operationhistory. Accordingly, the security of the personal information of theuser is improved.

Result information indicating a permission or non-permission of theaccess is transmitted from the multifunction machine B and the PC A tothe terminal device 10 (S03). When the access to the multifunctionmachine B and the PC A is permitted, a communication between theterminal device 10 and the multifunction machine B is established, andfurther, a communication between the terminal device 10 and the PC A isestablished.

Next, in the terminal device 10, the user instructs an execution of alinkage function (for example, the “printing function”) (S04). Accordingto the instruction, the terminal device 10 transmits informationindicating the instruction to execute the linkage function (executioninstruction information) to each of the devices to be linked (forexample, the multifunction machine B and the PC A) (S05). The control ofthe transmission may be performed by the automatic response AI. Theexecution instruction information transmitted to the multifunctionmachine B includes information indicating a process to be executed bythe multifunction machine B (for example, job information), and theexecution instruction information transmitted to the PC A includesinformation indicating a process to be executed by the PC A (forexample, job information).

Upon receiving the execution instruction information, the multifunctionmachine B and the PC A execute the linkage function (for example, the“printing function”) according to the execution instruction information(S06). When the linkage function to be executed includes a process oftransmitting and receiving data between the multifunction machine B andthe PC A, a communication between the multifunction machine B and the PCA is established. For example, the execution instruction informationtransmitted to the multifunction machine B includes the addressinformation of the PC A, and the execution instruction informationtransmitted to the PC A includes the address information of themultifunction machine B. The communication between the multifunctionmachine B and the PC A is established by using the address information.When the communication is established, the PC A transmits image data tobe printed to the multifunction machine B, and the multifunction machineB prints the image data on a paper which is a recording medium.

When the execution of the linkage function is completed, informationindicating the completion of the execution of the linkage function istransmitted from the multifunction machine B and the PC A to theterminal device 10 (S07). The UI unit 20 of the terminal device 10displays the information indicating the completion of the execution ofthe linkage function (S08). In addition, when the information indicatingthe completion of the execution is not displayed even after preset timeelapses from the time when the execution instruction is given, thecontroller 26 of the terminal device 10 may cause the UI unit 20 todisplay information indicating an error, and transmit the executioninstruction information or the information indicating the access requestto the multifunction machine B and the PC A again.

Next, the user confirms whether to release the linked state between themultifunction machine B and the PC A (S09), and a process correspondingto the confirmation result is executed (S10). When the linked state isreleased, the user gives a release instruction by using the terminaldevice 10. Accordingly, the communication between the terminal device 10and the multifunction machine B is disconnected, and the communicationbetween the terminal device 10 and the PC A is disconnected. Similarly,the communication between the multifunction machine B and the PC A isalso disconnected. When the linked state is not released, the executioninstruction may be continuously given.

In addition, the number of devices to be linked may be increased. Forexample, an image associated with a third device may be transmitted as aconversation content to the automatic response AI, and the three devicesincluding the multifunction machine B and the PC A may be linked to eachother. In addition, the information indicating that the multifunctionmachine B and the PC A have already been identified as devices to belinked is stored in the terminal device 10 or the server 16.

Device identification information indicating devices to be linked and alinkage function information indicating an executed linkage function maybe stored in, for example, the terminal device 10 or the server 16. Forexample, user account information (user identification information) ofthe user using the terminal device 10 may be acquired, and historyinformation may be created which indicates an association among the useraccount information, the device identification information indicatingdevices to be linked, and the linkage function information indicating anexecuted linkage function. The history information may be stored in, forexample, the terminal device 10 or the server 16. The historyinformation may be created by the terminal device 10 or the server 16.By referring to the history information, the linkage function that hasbeen executed and the device group that has been used for the linkagefunction are specified.

In addition, the devices to be linked (for example, the multifunctionmachine B and the PC A) may store the user account information of theuser making the access request or the terminal identificationinformation indicating the terminal device 10 which is an accessrequesting source, as history information. The user who has used thedevices is specified by referring to the history information. Forexample, when the user who is using a device needs to be specifiedbecause the device is broken down, or when a process of charging forconsumables or the like is performed, the user may be specified by usingthe history information. The history information may be stored in theterminal device 10, the server 16, or another device.

The user account information is stored in advance in, for example, thememory 24 of the terminal device 10, and the controller 26 of theterminal device 10 functions as an example of a user identification unitand identifies the user using the terminal device 10 by reading the useraccount information of the user from the memory 24. When the useraccount information of plural users is stored in the memory 24, the userdesignates his/her own account information by using the terminal device10. Accordingly, the user account information of the user is read, andthe user is identified. As another example, the controller 26 of theterminal device 10 may identify the user by reading user accountinformation of a user who is logging into the terminal device 10. As yetanother example, when only one piece of user account information isstored in the same terminal device 10, the controller 26 of the terminaldevice 10 may identify the user by reading the user account information.In addition, when no user account has been set and no user accountinformation has been created, initial setting is performed, andaccordingly, user account information is created.

When the user has been identified, for example, an image representingthe user himself/herself (for example, a user's own photo (for example,a face photo)) or an icon associated with the user rather than the imagerepresenting the user himself/herself may be displayed as the image 52associated with the user in FIG. 8. In addition, user identificationinformation such as a user name, a user ID, a handle name, or a nicknameof the identified user may be displayed in a character string on thescreen 48.

The use history of a linkage function may be managed for each user, andinformation indicating linkage functions used in the past by the userindicated by the read user account information may be displayed on theUI unit 20 of the terminal device 10. The information indicating the usehistory may be stored in the terminal device 10 or the server 16. Inaddition, information indicating a linkage function which has been usedwith frequency equal to or higher than preset use frequency may bedisplayed. By providing this shortcut function, the user's effort forthe operation related to a linkage function is reduced.

As described above, according to the first exemplary embodiment,information on a linkage function is provided when information ofdevices to be linked is transmitted as a conversation content of theuser to the automatic response AI, on the screen 48 for making aconversation with the automatic response AI. Thus, the information on alinkage function is provided by the simple method using a conversationwith the automatic response AI.

In addition, when three or more device images are transmitted as aconversation content to the automatic response AI, information on alinkage function executable by using the three or more devices isprovided.

In addition, when information of plural devices is transmitted as amessage content of the user to the automatic response AI within apredetermined time limit, the specifying unit 28 may specify a linkagefunction executable by using the plural devices, and when information ofa single device is transmitted as a message content of the user to theautomatic response AI within the time limit, the specifying unit 28 mayspecify a function (a solo function) that the device has. In this case,information on the linkage function or the solo function specified bythe specifying unit 28 is displayed on the display unit of the UI unit20. A starting time point of the time limit may be, for example, a timepoint when the automatic response AI requests the user to inform aboutdevices to be linked as in the message content 54, or a time point whenthe user transmits a first device image to the automatic response AIafter the message content 54 occurs.

For example, in the example illustrated in FIG. 8, in a case where theuser transmits only the device image 56 as a message content to theautomatic response AI within the time limit from the time point when theautomatic response AI requests the user to inform devices to be linkedas in the message content 54, the specifying unit 28 specifies afunction (a solo function) associated with the multifunction machine Bin the device function management table illustrated in FIG. 6. Thecontroller 26 displays the information on the solo function as a messagecontent of the automatic response AI in the display area 48 a. In thiscase, similarly to the process illustrated in FIG. 10, the informationindicating an access request is transmitted to the multifunction machineB, and when the access is permitted, the solo function designated by theuser is executed by the multifunction machine B.

Meanwhile, in a case where the user transmits the device images 56 and60 as a message content to the automatic response AI within the timelimit from the time point when the automatic response AI requests theuser to inform devices to be linked as in the message content 54, thespecifying unit 28 specifies linkage functions associated with thecombination of the multifunction machine B and the PC A in the linkagefunction management table illustrated in FIG. 7. The controller 26displays the information on the linkage functions in the display area 48a.

The similar process described above is also performed in a case wherethe starting time point of the time limit is the time point when theuser transmits a first device image (for example, the device image 56)to the automatic response AI.

By providing the time limit as described above, a switching betweenproviding a linkage function and providing a solo function is possible.

In the example illustrated in FIG. 8, when the user captures a device(for example, the multifunction machine B) by the camera 22 after theautomatic response AI requests the user to inform devices to be linkedas in the message content 54, the controller 26 transmits the deviceimage 56 associated with the device (the multifunction machine B) (forexample, an image obtained by the capturing or a schematic image) as amessage content of the user to the automatic response AI. The specifyingunit 28 identifies the multifunction machine B associated with thedevice image 56 as a device to be linked. Further, when the usercaptures another device (for example, the PC A) by the camera 22, thecontroller 26 transmits the device image 60 associated with the device(the PC A) as a message content of the user to the automatic responseAI. The specifying unit 28 identifies the PC A associated with thedevice image 60 as a device to be linked. In this case, the specifyingunit 28 specifies linkage functions associated with the combination ofthe multifunction machine B and the PC A in the linkage functionmanagement table illustrated in FIG. 7. The information on the linkagefunctions is displayed in the display area 48 a as a message content ofthe automatic response AI. In addition, the capturing of a device to belinked may be performed by an image capturing device other than thecamera 22. In this case, the device image data generated by thecapturing by the image capturing device is transmitted to the terminaldevice 10 via a communication path such as a network, and the deviceimage is transmitted as a message content of the user to the automaticresponse AI.

In addition, when a linkage function associated with the combination ofthe plural devices designated by the user is not registered in thelinkage function management table, the specifying unit 28 determinesthat the usable linkage function is not registered. In this case, thecontroller 26 displays a message indicating that the usable linkagefunction is not registered, in the display area 48 a as a messagecontent of the automatic response AI. Even in this case, the linkagefunction may become usable according to, for example, an updating statusof a device. In this case, the specifying unit 28 specifies the linkagefunction that becomes usable.

(Modification 1)

Hereinafter, Modification 1 will be described with reference to FIG. 13.FIG. 13 illustrates the screen 48. In Modification 1, when informationof plural devices to be linked is transmitted as a message content ofthe user to the automatic response AI, a linkage function executable byusing the plural devices is notified. Hereinafter, Modification 1 willbe described in detail.

As in FIG. 8, the screen 48 displays the image 50 associated with theautomatic response AI and the image 52 associated with the user, andfurther, displays information indicating a conversation between the userand the automatic response AI.

As in FIG. 8, the message content that “Which device do you want touse?” is displayed in the display area 48 a as a message content 80 ofthe automatic response AI.

When the user designates and transmits an image 82 representing pluraldevices as a message content of the user to the automatic response AI inresponse to the above described message content 80, the specifying unit28 identifies the plural devices represented in the image 82 as devicesto be linked. For example, when the image 82 represents the device image56 associated with the multifunction machine B and the device image 60associated with the PC A, the specifying unit 28 identifies themultifunction machine B as a device to be linked based on the deviceimage 56, and identifies the PC A as a device to be linked based on thedevice image 60.

The user may select the device images 56 and 60 from the stock area 58and superimpose the device images 56 and 60 on the image 82, byoperating the screen 48. Accordingly, the device images 56 and 60 arerepresented in the image 82, and the image 82 is transmitted as amessage content of the user to the automatic response AI.

As another example, the image 82 may be generated by capturing themultifunction machine B and the PC A together, by an image capturingdevice such as the camera 22. In addition, the user may capture themultifunction machine B and the PC A together by using the camera 22,and transmit the image 82 representing the multifunction machine B andthe PC A together as a message content of the user to the automaticresponse AI, after the message content 80 is made.

As described above, when the multifunction machine B and the PC A areidentified, the controller 26 displays the information of the devicesidentified in the display area 48 a as devices to be linked, as amessage content 84 of the automatic response AI. When the user approvesthe message content 84 (for example, when the user makes a messagecontent 86 answering “Yes”), the information on linkage functionsexecutable by using the PC A and the multifunction machine B isdisplayed in the display area 48 a as a message content 88 of theautomatic response AI. When the user designates a linkage function andgives an instruction to execute the linkage function by making a messagecontent 90 in response to the message content 88, the controller 26receives the execution instruction and transmits the executioninstruction information to the devices to be linked (the PC A and themultifunction machine B). Accordingly, the linkage function is executedby the PC A and the multifunction machine B. In addition, a screen forcontrolling the devices to be linked (a remote control screen) may bedisplayed on the display unit of the UI unit 20.

In addition, when three or more device images are represented in theimage 82, information on a linkage function executable by using thethree or more devices is notified. In addition, when one device image isrepresented in the image 82, information on a function (a solo function)that the device associated with the one device image has is notified.

In addition, when a device image associated with a device to be linkedis selected from the stock area 58, the information on a linkagefunction or a solo function may be notified under the control using thetime limit described above. For example, when plural device images aretaken out from the stock area 58 and superimposed on the image 82 withinthe time limit, the specifying unit 28 specifies a linkage functionexecutable by using the devices associated with the respective pluraldevice images. In this case, the information on the linkage function isdisplayed in the display area 48 a as a message content of the automaticresponse AI. Meanwhile, when one device image is taken out from thestock area 58 and superimposed on the image 82 within the time limit,the specifying unit 28 specifies a function (a solo function) that thedevice associated with the device image has. In this case, theinformation on the solo function is displayed in the display area 48 aas a message content of the automatic response AI.

For example, in the example illustrated in FIG. 13, when the userselects the device image 56 associated with the multifunction machine Bfrom the stock area 58 and superimposes the device image 56 on the image82, without superimposing another device image on the image 82, withinthe time limit, the specifying unit 28 specifies a function (a solofunction) associated with the multifunction machine B in the linkagefunction management table illustrated in FIG. 6. The controller 26displays the information on the solo function in the display area 48 aas a message content of the automatic response AI.

Meanwhile, when the user selects the device images 56 and 60 from thestock area 58 and superimposes the device images 56 and 60 on the image82 within the time limit, the specifying unit 28 specifies linkagefunctions associated with the combination of the multifunction machine Band the PC A in the linkage function management table illustrated inFIG. 7. The controller 26 displays the information on the linkagefunctions in the display area 48 a as a message content of the automaticresponse AI.

(Modification 2)

Hereinafter, Modification 2 will be described. In Modification 2, whenplural device images are transmitted as a message content of the user tothe automatic response AI, the controller 26 may generate a newcomposite image representing a state where the plural device images aresuperimposed on each other, or a state where the plural device imagesare adjacent and connected to each other. FIG. 14A illustrates anexample of the composite image. For example, when the device images 56and 60 are transmitted in this order as a message content of the user tothe automatic response AI, the controller 26 generates a new compositeimage 92 by grouping the device images 56 and 60. The controller 26 maydisplay the composite image 92 on the display unit of the UI unit 20 ormay register the composite image 92 as an image to be displayed in thestock area 58, in the terminal device 10. The composite image 92 may bean image representing the state where the device image 60 issuperimposed on the device image 56 or an image schematicallyrepresenting the state (that is, an icon).

When the device images 60 and 56 are transmitted in this order as amessage content of the user to the automatic response AI, the controller26 generates a composite image representing a state where the deviceimage 56 is superimposed on the device image 60.

In addition, as illustrated in FIG. 14B, the controller 26 may generatea composite image 94 representing a state where the device images 56 and60 are adjacent and connected to each other. The controller 26 maydisplay the composite image 94 on the display unit of the UI unit 20 ormay register the composite image 94 as an image to be displayed in thestock area 58, in the terminal device 10.

In addition, when plural device images are selected from the stock area58 and superimposed on the image 82 as in Modification 1, the controller26 may register the image 82 representing the plural device images as anew composite image in the terminal device 10.

When the composite image is generated as described above, the user maytransmit the composite image as a message content of the user to theautomatic response AI. For example, when the user transmits thecomposite image and another device image as a message content of theuser to the automatic response AI, the specifying unit 28 identifies theplural devices associated with the composite image and the deviceassociated with the other device image as devices to be linked. Forexample, when the user transmits the composite image 92 and the deviceimage associated with the projector C as a message content of the userto the automatic response AI, the specifying unit 28 identifies themultifunction machine B and the PC A which are associated with thecomposite image 92, as devices to be linked, and further, identifies theprojector C associated with the other device image as a device to belinked. In this case, the specifying unit 28 specifies a linkagefunction associated with the combination of the PC A, the multifunctionmachine B, and the projector C in the linkage function management tableillustrated in FIG. 7. In addition, the information on the linkagefunction is displayed as a message content of the automatic response AIon the screen 48. In addition, a new composite image in which thecomposite image 92 and the device image associated with the projector Care superimposed on each other, or a new composite image in which thecomposite image 92 and the device image associated with the projector Care adjacent and connected to each other may be generated.

As described above, the number of devices to be linked may be increasedby the simple operation using the composite image. Further, since thecomposite image represents plural devices to be linked, the user mayvisually grasp which devices are designated as devices to be linked, bydisplaying the composite image.

The composite image described above may be used as a shortcut image whenthe linkage function is executed next time and thereafter. For example,when the user gives an instruction to register the composite image as ashortcut image by using the terminal device 10, a registration screen isdisplayed on the display unit of the UI unit 20. The registration screendisplays the composite image, and displays information indicating theregistered contents of the linkage function executable by using theplural devices to be linked. For example, information indicating thelinkage function name, an outline of the linkage function, and devicesrequired for the linkage function is displayed as the informationindicating the registered contents. The user may edit the registeredcontents (for example, the linkage function name and an outline of thelinkage function) by using the terminal device 10. When the user givesthe registration instruction on the registration screen, the compositeimage is registered as a shortcut image. In addition, when the useredits the registered contents, the edited contents are also registeredin association with the composite image. For example, the shortcut imagewhich is the composite image is generated and displayed in the stockarea 58. The shortcut image is generated by, for example, the controller26 of the terminal device 10. When plural shortcut images are generatedand registered, the screen 48 may display a list of the shortcut images.In addition, the shortcut image and the information indicating theregistered contents are stored in the terminal device 10. As anotherexample, the shortcut image and the information indicating theregistered contents may be stored in the server 16 in association withthe user account information. In this case, when the user accountinformation is transmitted from the terminal device 10 to the server 16,the information associated with the user account information (theshortcut image and the information indicating the registered contents)is transmitted from the server 16 to the terminal device 10 anddisplayed in the terminal device 10.

In addition, the shortcut image may include information (for example, animage) indicating a status of each device to be linked. For example,when each device to be linked is in a state of being able to execute thelinkage function, an image indicating the state (for example, a green orblue image) is displayed as the information indicating the status. Wheneach device to be linked is in a state of being unable to execute thelinkage function, an image indicating the state (for example, a white orred image) is displayed as the information indicating the status. Thestate where the device is able to execute the linkage function refersto, for example, a state where the device is not being currently used ora state where the device is not broken down. The state where the deviceis unable to execute the linkage function refers to, for example, astate where the device is being currently used or a state where thedevice is broken down. For example, the automatic response AI acquiresthe information indicating the status of each device to be linked fromthe device to be linked, and displays the status information based onthe acquired information. The acquisition of the information may beperformed by the server 16.

When the user transmits the shortcut image as a message content of theuser to the automatic response AI, the information on the linkagefunction associated with the shortcut image may be displayed, or theexecution of the linkage function may be controlled.

As described above, the linkage function and the shortcut image aremanaged in association with each other, so that the user's burden toperform the operation to execute the linkage function next time andthereafter is lessened.

(Modification 3)

Hereinafter, Modification 3 will be described. In Modification 3, thenotification of a linkage function is controlled according to an orderof transmitting information of a device to the automatic response AI.Hereinafter, Modification 3 will be described in detail with referenceto FIGS. 15 to 17.

FIG. 15 illustrates an example of a linkage function management tableaccording to Modification 3. In the linkage function management table,for example, information indicating a combination of device IDs,information indicating a device name of each device to be linked (forexample, a type of each device), information indicating a linkagefunction (linkage function information), information indicating anaccess order, and information indicating a priority are associated witheach other. The device ID and the device name correspond to an exampleof the device identification information. In addition, a linkagefunction executable by using three or more devices may be set. In thiscase, a combination of the three or more devices and a linkage functionare associated with each other, and the association is registered in thelinkage function management table.

The access order corresponds to an order of designating a device, andfor example, corresponds to an order of transmitting information of adevice (for example, a device image, character information, or voiceinformation) as a message content of the user to the automatic responseAI. For example, when the user sequentially transmits information ofeach of plural devices as a message content of the user to the automaticresponse AI, the sequential order corresponds to the access order.Specifically, when the user first transmits information of a firstdevice as a message content of the user to the automatic response AI,and then, transmits information of a second device as a message contentof the user to the automatic response AI, the first device is considereddesignated first, and the second device is considered designated second.

The priority corresponds to a priority for the notification (forexample, display or voice output) of information on a linkage function.For example, the combination of the PC A and the multifunction machine Bis associated with the “scanning transmitting function” and the“printing function” which are linkage functions. For example, when theinformation of each of the multifunction machine B and the PC A istransmitted in this order as a message content of the user to theautomatic response AI, the “scanning transmitting function” has a “firstpriority,” and the “printing function” has a “second priority.” In thiscase, the information on the “scanning transmitting function” isnotified in preference to the information on the “printing function.”For example, the information on the “scanning transmitting function” isdisplayed at a higher rank or earlier than the information on the“printing function.” Meanwhile, when the information of each of the PC Aand the multifunction machine B is transmitted in this order as amessage content of the user to the automatic response AI, the “printingfunction” has a “first priority,” and the “scanning transmittingfunction” has a “second priority.” In this case, the information on the“printing function” is displayed in preference to the information on the“scanning transmitting function.”

FIG. 16 illustrates an example of the display according to the priority.For example, when the device images 56 and 60 are transmitted in anorder of the multifunction machine B and the PC A as a message contentof the user to the automatic response AI, the “scanning transmittingfunction” has a “first priority,” and the “printing function” has a“second priority.” Thus, the information on the “scanning transmittingfunction” is displayed in preference to (for example, at a higher rank)the information on the “printing function” on the screen 48. When theuser designates a linkage function and gives an execution instruction ofthe linkage function on the screen 48, the designated linkage functionis executed.

FIG. 17 illustrates another example of the display according to thepriority. For example, when the device images 56 and 60 are transmittedin an order of the PC A and the multifunction machine B as a messagecontent of the user to the automatic response AI, the “printingfunction” has a “first priority,” and the “scanning transmittingfunction” has a “second priority.” Thus, the information on the“printing function” is displayed (for example, at a higher rank) inpreference to the information on the “scanning transmitting function” onthe screen 48.

As described above, the order of notifying information on a linkagefunction (for example, a displaying order) is changed according to theorder of transmitting information of a device as a message content ofthe user to the automatic response AI. The access order of a device (theorder of transmitting information of a device to the automatic responseAI) also serves as an order of using a function of each device or anorder of moving data between linked devices. Further, the operation todesignate a device also serves as an operation to designate the order ofusing a function or the order of moving data. Thus, information on alinkage function which is predicted to be used by the user ispreferentially notified, by changing the order of notifying informationof a linkage function according to the access order. That is,information on a linkage function which is highly likely to be used bythe user is preferentially notified. For example, when information ofthe multifunction machine B and the PC A is transmitted in this order asa message content of the user to the automatic response AI, it ispredicted that the user is to use a linkage function to “transmit datafrom the multifunction machine B to the PC A by using the function ofthe multifunction machine B in preference to the PC A.” In addition,when information of the PC A and the multifunction machine B istransmitted in this order as a message content of the user to theautomatic response AI, it is predicted that the user is to use a linkagefunction of “transmitting data from the PC A to the multifunctionmachine B by using the function of the PC A in preference to themultifunction machine B.” Thus, information on a linkage function whichis highly likely to be used by the user is preferentially notified, bychanging the order of notifying information on a linkage functionaccording to the order of selecting a device.

(Modification 4)

Hereinafter, Modification 4 will be described. In Modification 4,different functions are assigned to respective portions in a deviceimage associated with a device. When portions in a device image aredesignated by the user, and information of the portions is transmittedas a message content of the user to the automatic response AI,information on a linkage function executable by using the functionsassigned to the portions is notified. Hereinafter, Modification 4 willbe described in detail.

FIG. 18 illustrates an example of a device function management tableaccording to Modification 4. In the device function management table,for example, a device ID, information indicating a device name (forexample, a type of a device), information indicating a position in adevice image (a position of a portion), and information indicating afunction corresponding to a position in a device image (a position of aportion) (function information), and an image ID are associated witheach other. The position in a device image is a specific position (aspecific portion) in a device image associated with a device and is, forexample, a specific position in a device image schematicallyrepresenting a device or a specific position in a device image capturedby the camera. The respective specific positions (portions) in a deviceimage are associated with different functions. In addition, a linkagefunction executable by using three or more portions may be set. In thiscase, the combination of the three or more portions and a linkagefunction are associated with each other, and the association isregistered in the linkage function management table. In addition, alinkage function executable by using a portion and an entire device maybe set, and the association between the combination of the portion andthe entire device and the linkage function may be registered in thelinkage function management table.

Hereinafter, portions of a device will be described in detail withreference to FIGS. 19A, 19B, 20A, and 20B. Each of FIGS. 19A and 19Billustrates a device image 96 associated with the multifunction machineB as a device. Each of FIGS. 20A and 20B illustrates a device image 98associated with the PC A as a device. The multifunction machine B or thePC A is identified by applying the identification technique as describedabove (for example, the AR technique). In addition, the identificationof a portion is also implemented by using the identification techniquedescribed above. For example, each portion of a device is identified bycapturing each portion and using the markerless AR technique.

For example, as illustrated in FIG. 19A, the “printing function” isassigned to the position associated with the main body portion of themultifunction machine B (a portion image 96 a) in the device image 96.The “scanning function” is assigned to the position associated with thedocument cover, the document glass, or the automatic document feeder ofthe multifunction machine B (a portion image 96 b) in the device image96. A “stapling function” is assigned to the position associated withthe post-processing device (a portion image 96 c) in the device image96. The “stapling function” is a function of binding output papers witha staple.

In addition, as illustrated in FIG. 20A, a “data saving function” isassigned to the position associated with the main body portion of the PCA (a portion image 98 a) in the device image 98. A “screen displayingfunction” is assigned to the specific position associated with thedisplay unit of the PC A (a portion image 98 b) in the device image 98.The “data saving function” is a function of saving data transmitted fromanother device in the PC A. The “screen displaying function” is afunction of displaying data transmitted from another device by the PC A.

Each portion image may be operated to be separated from the deviceimage. For example, when the device image 96 is displayed on the displayunit of the UI unit 20, and the user selects the portion image 96 b andperform an operation to separate the portion image 96 b from the deviceimage 96, the portion image 96 b is displayed apart from the deviceimage 96 as illustrated in FIG. 19B. In FIG. 19B, the separated portionimage 96 b is indicated by a dashed line. For example, the portion imagemay be operated by the dragging and dropping operations. Specifically,the user may select a portion image, separate the portion image from thedevice image by performing the dragging operation, and display theportion image at a desired display position by performing the droppingoperation at the display position. In addition, when the device image 96is displayed in the stock area 58, and the user selects a portion imageof the device image 96 (for example, the portion image 96 b) andperforms an operation to separate the portion image from the deviceimage 96, the portion image 96 b may be displayed apart from the deviceimage 96 outside the stock area 58. In addition, even though theoperation to separate the portion image from the device image has beenperformed, the portion image may be still displayed at the originalposition. The portion images 96 a and 96 c may also be displayed to beseparated from the device image 96.

In the device image 98 as well, the portion images 98 a and 98 b may bedisplayed to be separated from the device image 98, as in the deviceimage 96. In FIG. 20B, the portion image 98 b (indicated by a dashedline) is separated from the device image 98.

In Modification 4, the user may transmit portion images as a messagecontent of the user to the automatic response AI. In this case, thespecifying unit 28 identifies the portions associated with the portionimages as portions to be linked. When the user transmits plural portionimages as a message content of the user to the automatic response AI,the specifying unit 28 identifies the portions associated with therespective plural portion images as portions to be linked, and specifiesa linkage function executable by using the plural portions.

In addition, when a device image is displayed on the display unit of theUI unit 20, the controller 26 may display the name of a functionassigned to each position in the device image (for example, printing orscanning) on the display unit of the UI unit 20. Accordingly, theinformation of the function assigned to each position is provided to theuser. In addition, the name of the function may not be displayed.

Hereinafter, a process using portion images will be described by usingspecific examples.

FIG. 21 illustrates the screen 48. As in FIG. 8, the screen 48 displaysthe image 50 associated with the automatic response AI and the image 52associated with the user, and further, displays information indicating aconversation between the user and the automatic response AI. Further,the screen 48 displays the stock area 58.

As in FIG. 8, the message content that “Which device do you want touse?” is displayed in the display area 48 a as the message content 54 ofthe automatic response AI.

When the user designates portions of devices in response to the messagecontent 54, the specifying unit 28 (the specifying unit 28 implementedby the automatic response AI) identifies the portions of the devices asportions to be linked. For example, as illustrated in FIG. 21, when theuser designates the portion image 96 b associated with a portion of themultifunction machine B and transmits the portion image 96 b as amessage content of the user to the automatic response AI, the specifyingunit 28 identifies the portion associated with the portion image 96 b asa portion to be linked. In the example illustrated in FIG. 21, theoperation by the user to display the portion image 96 b in the displayarea 48 b corresponds to an act for the user to make a message contentas in the example illustrated in FIG. 8. In addition, the user may inputa character string indicating the portion of the device as a messagecontent of the user to the terminal device 10 by operating the UI unit20. In this case, the specifying unit 28 identifies the portionindicated by the character string as a portion to be linked. Inaddition, the information of the portion to be linked may be input byvoice as a message content of the user to the terminal device 10.

For example, the user selects the portion image 96 b included in thedevice image 96 displayed in the stock area 58, moves the portion image96 b from the stock area 58 to the position where a message content ofthe user is displayed, by the dragging operation, and performs thedropping operation at the position where a message content is displayed.With the operations, the controller 26 transmits the portion image 96 bas a message content of the user to the automatic response AI.

In the example illustrated in FIG. 21, in addition to the portion image96 b, the user takes the portion image 98 b out from the stock area 58,and moves the portion image 98 b to the position where a message contentof the user is displayed. With the operation, the portion image 98 b istransmitted as a message content of the user to the automatic responseAI.

When the user transmits the information of the plural portions as amessage content of the user to the automatic response AI on the screen48, that is, when the user makes a message indicating the information ofthe plural portions as information of portions to be linked, thespecifying unit 28 identifies the plural portions as portions to belinked. In the example described above, the multifunction machine Bassociated with the device image 96 including the portion image 96 b andthe PC A associated with the device image 98 including the portion image98 b are identified as devices to be linked, and the “scanning function”corresponding to the portion image 96 b and the “screen displayingfunction” corresponding to the portion image 98 b are identified asfunctions to be linked.

When the devices to be linked (for example, the PC A and themultifunction machine B) are identified, the specifying unit 28specifies linkage functions associated with the combination of the PC Aand the multifunction machine B in the linkage function management tableillustrated in FIG. 7. Further, the specifying unit 28 specifiesfunctions associated with the portions in the device images transmittedas a message content of the user to the automatic response AI, byreferring to the device function management table illustrated in FIG.18. Then, the specifying unit 28 raises the priority of the linkagefunction which uses the functions associated with the portions, andlowers the priority of the linkage function which does not use thefunctions, among the linkage functions executable by using the PC A andthe multifunction machine B. The controller 26 of the terminal device 10displays the information on the linkage functions according to thepriorities as a message content of the automatic response AI on thescreen 48.

For example, when the multifunction machine B and the PC A areidentified as devices to be linked, and the portions of the devices areidentified as portions to be linked, the controller 26 display theinformation of the portions of the devices identified as devices to belinked as a message content 100 of the automatic response AI on thescreen 48. When the user approves the message content 100 (for example,when the user makes a message content 102 answering “Yes”), the screen48 displays the information on the linkage functions executable by usingthe plural portions designated by the user.

FIG. 22 illustrates the screen 48 which is continuous to FIG. 21. Whenthe linkage functions are specified as described above, the controller26 display the information on the linkage functions in the display area48 a as a message content 104 of the automatic response AI. Since the“scanning function” and the “screen displaying function” are designatedin this order by the user, information on a “scanning transmittingdisplaying function” which is a linkage function executable by using the“scanning function” and the “screen displaying function” is displayed inpreference to (for example, at a higher rank) information on the otherlinkage functions. For example, the information on the “scanningtransmitting displaying function” is displayed in preference toinformation on a “scanning transmitting saving function” which is alinkage function executable by using the “scanning function” and “datasaving function.” In addition, the “scanning transmitting displayingfunction” is a function of transmitting data generated by scanning bythe multifunction machine B to the PC A and display the data on thescreen of the PC A. The “scanning transmitting saving function” is afunction of transmitting data generated by scanning by the multifunctionmachine B to the PC A and save the data in the PC A.

When the user designates a linkage function by a message content 106 andgives an instruction to execute the linkage function in response to themessage content 104 of the automatic response AI, the controller 26receives the execution instruction and transmits the executioninstruction information to the devices to be linked. Further, thecontroller 26 displays information indicating that the executioninstruction has been received, in the display area 48 a as a messagecontent 108 of the automatic response AI. For example, when aninstruction to execute the “scanning transmitting displaying function”is given, data generated by scanning by the multifunction machine B istransmitted to the PC A and displayed on the screen of the PC A.

In addition, a screen for controlling the devices to be linked (forexample, the multifunction machine B and the PC A) (a remote controlscreen) may be displayed on the display unit of the UI unit 20.

According to Modification 4, when devices to be linked have pluralfunctions, the functions are individually designated, and information ona linkage function executable by using the designated functions ispreferentially notified. Thus, a linkage function which is predicted tobe used by the user is preferentially notified.

In addition, the linkage function may be a function using a combinationof portions of the same device or portions of different devices, afunction using a combination of an entire device and a portion of adevice, or a function using a combination of entire devices.

In addition, by combining modifications 3 and 4, the display ofinformation on a linkage function may be switched according to an orderof transmitting information of a device or a portion of a device as amessage content of the user to the automatic response AI.

In addition, when information of one portion is transmitted as a messagecontent of the user to the automatic response AI, information on a solofunction executable by using the one portion is displayed on the displayunit of the UI unit 20.

Hereinafter, another example of the process using portion images will bedescribed.

FIG. 23 illustrates another example of the device function managementtable according to Modification 4. In the device function managementtable, for example, a device ID, information indicating a device name(for example, a type of a device), information indicating a name of aportion of a device (for example, a type of the portion), a portion IDwhich is portion identification information for identifying the portion,information indicating a function assigned to the portion (a functionthat the portion has), and a portion image ID for identifying a portionimage associated with the portion are associated with each other. Theportion image represents the external appearance of a portion of adevice obtained by capturing with the camera. In addition, a portionimage schematically representing a portion of a device may be associatedwith the portion. For example, different functions are assigned torespective portions of a device.

As a specific example, the screen displaying function is assigned to thedisplay portion of the PC A, and information indicating the screendisplaying function is associated with the portion image ID of theportion image associated with the display portion. The screen displayingfunction is a function of displaying information by the PC A. The datasaving function is assigned to the main body portion of the PC A, andinformation indicating the data saving function is associated with theportion image ID of the portion image associated with the main bodyportion. The data saving function is a function of saving data in the PCA.

In addition, the printing function is assigned to the main body portionof the multifunction machine B, and information indicating the printingfunction is associated with the portion image ID of the portion imageassociated with the main body. The scanning function is assigned to thereading unit of the multifunction machine B (for example, the portionassociated with the document cover, the document glass, or the automaticdocument feeder of the multifunction machine B), and informationindicating the scanning function is associated with the portion image IDof the portion image associated with the reading portion. The staplingfunction is assigned to the post-processing device of the multifunctionmachine B, and information indicating the stapling function isassociated with the portion image ID of the portion image associatedwith the post-processing device. The stapling function is a function ofbinding output papers with a staple.

The functions assigned to the portions of the devices are specified(identified) by using the markerless AR technique. For example, when aportion of a device is captured by a camera (for example, the camera 22of the terminal device 10), the specifying unit 28 specifies(identifies) a function associated with the external appearance imagedata in the device function management table illustrated in FIG. 23.Accordingly, the function assigned to the captured portion is specified(identified). For example, when the main body portion of themultifunction machine B is captured by the camera 22, the specifyingunit 28 specifies the printing function associated with the externalappearance image data representing the main body portion of themultifunction machine B in the device function management table.Accordingly, the printing function assigned to the main body portion ofthe multifunction machine B is specified.

In addition, the functions assigned to the portions of the devices maybe specified (identified) by using the marker AR technique. For example,each portion of a device is provided with a marker such as atwo-dimensional barcode obtained by encoding portion identificationinformation for identifying the portion (for example, the portion ID).When the marker provided in the portion is captured by a camera and themarker AR technique is applied, the portion identification information(for example, the portion ID) of the portion is acquired. When theportion identification information is acquired, the specifying unit 28specifies (identifies) a function associated with the portionidentification information (for example, the portion ID) in the devicefunction management table illustrated in FIG. 23.

FIG. 24 illustrates an example of a linkage function management tableaccording to Modification 4. The linkage function management table isinformation indicating a linkage function using functions that pluralportions have. In the linkage function management table, for example,information indicating a combination of portions of devices, informationindicating a combination of portion IDs, and information indicating alinkage function executable by using functions that the plural portionsincluded in the combination have are associated with each other. Inaddition, in the linkage function management table, informationindicating a combination of a portion of a device and an entire deviceand information indicating a linkage function using a function that theportion of the device has and a function that the entire device has maybe associated with each other.

As a specific example, the “printing function” is assigned as a linkagefunction to a combination of the display unit of the PC A and the mainbody portion of the multifunction machine B. The “printing function”which is a linkage function is a function of transmitting data saved inthe PC A to the multifunction machine B and print the data by themultifunction machine B.

Further, the “printing function” is assigned as a linkage function to acombination of the main body portion of the multifunction machine B andthe main body portion of the projector C. The “printing function” whichis a linkage function is a function of transmitting data projected bythe projector C to the multifunction machine B and print the data by themultifunction machine B.

Further, a “scanning projecting function” is assigned as a linkagefunction to a combination of the reading unit of the multifunctionmachine B and the main body portion of the projector C. The “scanningprojecting function” which is a linkage function is, for example, afunction of transmitting data generated by scanning by the multifunctionmachine B to the projector C and project the data by the projector C.

In addition, the linkage function may be a function executable by usingfunctions that plural portions included in the same device have, or afunction executable by using functions that portions of plural differentdevices have. In addition, the linkage function may be a functionexecutable by using functions that three or more portions have.

For example, when the user transmits plural portion images as a messagecontent of the user to the automatic response AI, the specifying unit 28specifies (identifies) a linkage function associated with a combinationof the portions associated with the respective plural portion images, inthe linkage function management table illustrated in FIG. 24. Forexample, when a portion image associated with the main body portion ofthe multifunction machine B and a portion image associated with the mainbody portion of the projector C are transmitted to the automaticresponse AI, the specifying unit 28 specifies, for example, the“printing function” which is the linkage function associated with thecombination of the main body portion of the multifunction machine B andthe main body portion of the projector C in the linkage functionmanagement table illustrated in FIG. 24. As illustrated in FIG. 22, theinformation on the linkage function is displayed in the display area 48a as a message content of the automatic response AI.

In addition, by combining modifications 3 and 4, the display ofinformation on a linkage function may be switched according to an orderof transmitting a device image or a portion image as a message contentof the user to the automatic response AI.

(Modification 5)

Hereinafter, Modification 5 will be described. In Modification 5,information on a linkage function is first input by the user as amessage content of the user to the terminal device 10, and then,information on devices to be linked is input as a message content of theuser to the terminal device 10. Hereinafter, Modification 5 will bedescribed in detail.

FIG. 25 illustrates the screen 48. As in FIG. 8, the screen 48 displaysthe image 50 associated with the automatic response AI, the image 52associated with the user, and information indicating a conversationbetween the user and the automatic response AI.

First, the user inputs a message content 110 indicating a request forexecuting a linkage function. In the example illustrated in FIG. 25, theuser requests a linkage function of transmitting scanned data to a PC.The automatic response AI determines that the user is requestingexecution of a linkage function, by analyzing the message content 110,and make a message content 112 for requesting the user to inform devicesto be used for the linkage function. The controller 26 displays themessage content 112 in the display area 48 a. In response to the messagecontent 112, the user inputs information on devices to be used for thelinkage function to the terminal device 10. The user may input theinformation on the devices to the terminal device 10 by placing deviceimages as the information on the devices at the position where a messagecontent of the user is displayed, or may input the information on thedevices to the terminal device 10 by voice. In the example illustratedin FIG. 25, the device images 56 and 60 are displayed at the positionwhere a message content of the user is displayed. Accordingly, thecontroller 26 transmits the device images 56 and 60 as a message contentof the user to the automatic response AI, and the specifying unit 28identifies the multifunction machine B associated with the device image56 and the PC A associated with the device image 60 as devices to belinked. The automatic response AI makes a message content 114 indicatingthe identified devices, and the controller 26 display the messagecontent 114 in the display area 48 a. When the user makes a messagecontent 116 indicating an instruction to execute the linkage function inresponse to the message content 114, the controller 26 causes themultifunction machine B and the PC A to execute the linkage functionrequested by the user. Further, the controller 26 display theinformation indicating that the instruction to execute the linkagefunction has been made, in the display area 48 a as a message content118 of the automatic response AI.

As described above, according to Modification 5, even when the userfirst transmits information on a linkage function to the automaticresponse AI, the linkage function may be executed by the simpleoperation through a conversation with the automatic response AI.

(Modification 6)

Hereinafter, Modification 6 will be described. In Modification 6, anotification of a linkage function is controlled according toinformation obtained by the automatic response AI by a learning functionof the automatic response AI. The automatic response AI acquires, forexample, information on the user (for example, information indicating anoperation history of the terminal device 10, information indicating anoperation history of a device, and information indicating a behaviorhistory), and information on the devices 12 and 14 (for example,information indicating a use history and information indicating a stateof each device), by the learning function. The automatic response AI mayacquire the information from the terminal device 10, devices such as thedevices 12 and 14, the server 16, another device (for example, a cloudsystem), or an operation assistant stored in another device (forexample, AI stored in another device).

The controller 26 controls a notification of a linkage functionaccording to the information acquired by the automatic response AI. Forexample, when the automatic response AI acquires information indicatinga history of use of a device by the user or information indicating theuser's schedule (plan) (schedule information), the controller 26 refersto the use history or the schedule, and preferentially displaysinformation on a linkage function which is highly likely to be used bythe user, as a message content of the automatic response AI on thedisplay unit of the UI unit 20. For example, when the user transmitsplural device images as a message content of the user to the automaticresponse AI, plural linkage functions executable by using the devicesassociated with the plural device images are specified. The controller26 analyzes the use history or the schedule, and preferentially displaysinformation on a linkage function which is highly likely to be used bythe user among the plural linkage functions, on the display unit of theUI unit 20. For example, the controller 26 compares timings andschedules in which the plural devices are designated as devices to belinked in the past, and preferentially displays information on a linkagefunction which is highly likely to be used at the present time, on thedisplay unit of the UI unit 20.

As a specific example, it is assumed that the user transmitted first andsecond device images as a message content of the user to the automaticresponse AI in the morning of a day, in the past, when the user went outin the afternoon, and a linkage function Z is executed. In thecircumstance where the use history described above exists and when theuser transmits the first and second device images as a message contentof the user to the automatic response AI at a timing during the morningof a day when the user plans to go out in the afternoon, the controller26 determines that the linkage function Z is highly likely to be used,and preferentially displays the information on the linkage function Z onthe display unit of the UI unit 20.

As another specific example, it is assumed that first and second deviceimages are transmitted as a message content of the user to the automaticresponse AI in the certain time period in the past, and the videorecording function is executed as a linkage function so that a programof a channel X is recorded. In the circumstance where the use historydescribed above exists, when the first and second device images aretransmitted as a message content of the user to the automatic responseAI at a timing included in the certain time period in the past, thecontroller 26 determines that the video recording function describedabove is highly likely to be used, and preferentially displays theinformation on the recording function (for example, information of thefunction of recording the channel X) on the display unit of the UI unit20.

In addition, when the devices associated with the device imagestransmitted to the automatic response AI are being used or broken down,the controller 26 may display the information on the linkage functionexecutable by using the devices, in a state where the priority of theinformation is lowered, on the display unit of the UI unit 20. When thedevices are not being broken down and are not being used, the controller26 preferentially displays the information on the linkage functionexecutable by using the devices on the display unit of the UI unit 20.Accordingly, the information on an immediately usable linkage functionis preferentially provided to the user.

(Modification 7)

Hereinafter, Modification 7 will be described. In Modification 7, theautomatic response AI recommends a device to be linked. For example,when the user transmits a device image as a message content of the userto the automatic response AI, the specifying unit 28 identifies thedevice associated with the device image, and specifies a device capableof executing a linkage function in combination with the identifieddevice by referring to the linkage function management table. Thecontroller 26 displays the information on the device specified by thespecifying unit 28 as a message content of the automatic response AI onthe display unit of the UI unit 20. For example, when the user transmitsthe device image 56 associated with the multifunction machine B as amessage content of the user to the automatic response AI, the specifyingunit 28 specifies a device capable of executing a linkage function incombination with the multifunction machine B (for example, a PC or aprojector), by referring to the linkage function management table. Theinformation on the specified device (for example, a PC or a projector)is displayed as a message content of the automatic response AI on thedisplay unit of the UI unit 20. Accordingly, the device to be linked isprovided to the user.

In addition, the specifying unit 28 may specify a device which isactually used in combination with the device associated with the deviceimage described above (for example, the multifunction machine B), basedon a use history of a linkage function (a history of use by the user oranother user), and the controller 26 may display the information on thespecified device as a message content of the automatic response AI onthe display unit of the UI unit 20. For example, the specifying unit 28may specify a device which is used in combination with the deviceassociated with the device image described above (for example, themultifunction machine B) with a frequency (for example, the number oftimes of use) equal to or more than a predetermined threshold.Accordingly, a device having a relatively high use frequency is providedto the user.

In addition, the controller 26 may preferentially display information ona device used with a relatively higher frequency in combination with thedevice associated with the device image described above (for example,the multifunction machine B), on the display unit of the UI unit 20.

(Modification 8)

Hereinafter, Modification 8 will be described. In Modification 8, pluralautomatic response AIs may be linked to each other. For example, theautomatic response AI stored in the terminal device 10 may shareinformation (for example, information of a use history of a linkagefunction and information of a use history of a device) with an automaticresponse AI stored in another terminal device. In this case, thespecifying unit 28 may specify a linkage function or recommend a deviceto be linked as in Modification 7, by using the shared information whichis acquired by the automatic response AI. For example, the specifyingunit 28 may specify a device capable of executing a linkage function incombination with a device transmitted by the user to the operationassistant, by using the information of the use history included in theshared information.

In addition, an automatic response AI may be set for each user. In thiscase, plural automatic response AIs may be stored in the same terminaldevice 10. Each automatic response AI may manage information on the userassociated with the own automatic response AI (for example, informationof an operation history of the terminal device 10, information of a usehistory of a linkage function, and information of a use history of adevice), and share the information with the other automatic responseAIs. In this case, each automatic response AI may recommend a device tobe linked or recommend a linkage function by using the sharedinformation.

(Modification 9)

Hereinafter, Modification 9 will be described. In Modification 9, aconversation between the user and the automatic response AI is conductedby voice. For example, when the user utters names of devices to belinked (for example, the multifunction machine B and the PC A) by voice,the voice is input to the terminal device 10, and the specifying unit 28(the specifying unit 28 implemented by the automatic response AI)receives the input of the voice and specifies a linkage function. Theinformation on the linkage function may be displayed as a messagecontent of the automatic response AI on the display unit of the UI unit20 or may be issued as voice information. When a voice input is used,the image associated with the automatic response AI may not be displayedon the display unit of the UI unit 20. In addition, when the imageassociated with the automatic response AI is displayed, the informationon the devices may be input by voice as a message content of the user tothe terminal device 10.

Second Exemplary Embodiment

Hereinafter, a device system according to a second exemplary embodimentwill be described. In the first exemplary embodiment described above,when information of devices is transmitted as a message content of theuser to the automatic response AI, information on a linkage function isnotified as a message content of the automatic response AI. Meanwhile,in the second exemplary embodiment, when information of functions istransmitted as a message content of the user to the automatic responseAI, information on a linkage function is notified as a message contentof the automatic response AI. The information of a function isidentification information for identifying the function and is, forexample, a function image associated with the function or characterinformation or voice information indicating the identificationinformation. For example, when the user transmits a function imageassociated with a function to the automatic response AI, when the userprovides the automatic response AI with a character string indicatingidentification information of a function by inputting the characterstring to the terminal device 10, or when the user provides theautomatic response AI with identification information of a function byvoice, the specifying unit 28 identifies the function.

The function image may be, for example, an image associated withsoftware (for example, an application) or an image associated with adevice by which a process is executed (a device existing in a realspace). The function image is, for example, an image such as an icon andmay be said as a virtual image which is distinguishable from an imagerepresenting the actual external appearance of a hardware device (forexample, a photo) or an image representing a device itself. For example,when function images associated with pieces of software are transmittedas a message content of the user to the automatic response AI,information indicating a linkage function executable by using the piecesof software associated with the function images is notified as a messagecontent of the automatic response AI. Hereinafter, the second exemplaryembodiment will be described in detail.

FIG. 26 illustrates an example of a linkage function management tableaccording to the second exemplary embodiment. The linkage functionmanagement table is information indicating a linkage function executableby using plural functions. In the linkage function management table, forexample, a function ID (for example, an ID for identifying software),information indicating a function name (for example, a software name),and information indicating a linkage function executable by using pluralfunctions are associated with each other. A linkage function executableby using plural functions is specified by referring to the linkagefunction management table.

For example, software α as a function is software for transmitting data,and software β as a function is software for setting a password fordata. For example, a linkage function of setting a password for data andthen transmitting the data is implemented by combining the software αand the software β. Executable linkage functions are also set for otherpieces of software.

In addition, a linkage function executable by using three or morefunctions (for example, pieces of software) may be set. In this case,the combination of the three or more functions and the linkage functionare associated with each other, and the association is registered in thelinkage function management table.

Hereinafter, the second exemplary embodiment will be described in detailby using specific examples.

FIG. 27 illustrates the screen 48. As in FIG. 8, the screen 48 displaysthe image 50 associated with the automatic response AI, the image 52associated with the user, and information indicating a conversationbetween the user and the automatic response AI.

For example, a message content 120 that “Which function do you want touse?” is displayed in the display area 48 a as a message content of theautomatic response AI.

When the user designates functions in response to the message content120, the specifying unit 28 (the specifying unit 28 implemented by theautomatic response AI) identifies the functions as functions to belinked. For example, as illustrated in FIG. 27, when the user designatesa function image 122 associated with a function and places the functionimage 122 at the position where a message content of the user isdisplayed, the controller 26 transmits the function image 122 as amessage content of the user to the automatic response AI. In response tothis operation, the specifying unit 28 identifies the functionassociated with the function image 122 as a function to be linked. Inthe example illustrated in FIG. 27, the operation by the user to displaythe function image 122 in the display area 48 b corresponds to an actfor the user to make a message content. In addition, the user may inputa character string indicating a function as a message content of theuser to the terminal device 10 by operating the UI unit 20. In thiscase, the specifying unit 28 identifies the function indicated by thecharacter string as a function to be linked. In addition, information ofa function to be linked may be input by voice as a message content ofthe user to the terminal device 10.

The function image 122 is an image associated with a password settingfunction (for example, an image associated with password settingsoftware). When the user selects the function image 122 and makes aninstruction to activate the password setting function, for example, thepassword setting software is activated, and the password setting processis performed on, for example, data.

For example, the controller 26 displays a stock area 124 in the displayarea 48 b for the user, and displays function images in the stock area124, as in the first exemplary embodiment. For example, the controller26 displays function images associated with software installed in theterminal device 10 in the stock area 124. In addition, when there existsanother function image that may not be displayed in the stock area 124,the other function image is displayed in the stock area 124 in themanner that the user performs an operation to scroll the function imagesdisplayed in the stock area 124.

As in the first exemplary embodiment, the user may select a functionimage associated with a function to be linked (the function image 122 inthe example illustrated in FIG. 27) from the stock area 124, move thefunction image to the outside of the stock area 124, and transmit thefunction image 122 as a message content of the user to the automaticresponse AI. For example, the user moves the function image 122 from thestock area 124 to the position where a message content of the user isdisplayed, by performing the dragging operation, and performs thedropping operation at the position where a message content is displayed.With the operations, the controller 26 transmits the function image 122as a message content of the user to the automatic response AI.

In the example illustrated in FIG. 27, in addition to the function image122, the user selects a function image 126 from the stock area 124 andmoves the function image 126 to the position where a message content ofthe user is displayed. With the operation, the function image 126 istransmitted as a message content of the user to the automatic responseAI. The function image 126 is an image associated with a datatransmitting function (for example, an image associated with datatransmitting software). When the user selects the function image 126 andmakes an instruction to activate the data transmitting function, forexample, the data transmitting software is activated, and data or thelike to be transmitted is transmitted to a transmission destination.

When the user transmits the information of the plural functions to theautomatic response AI on the screen 48, that is, when the user makes amessage indicating the information of the plural functions asinformation of functions to be linked, the specifying unit 28 specifiesa linkage function associated with the plural functions in the linkagefunction management table illustrated in FIG. 26. The controller 26displays the information on the linkage functions in the display area 48a as a message content of the automatic response AI.

In the example described above, the function images 122 and 126 aretransmitted as functions to be linked to the automatic response AI. Inthis case, the specifying unit 28 identifies the function associatedwith the function image 122 and the function associated with thefunction image 126 as functions to be linked. In addition, the functionimage 122 may be first transmitted to the automatic response AI, andthen, the function image 126 may be transmitted to the automaticresponse AI, or the function image 126 may be first transmitted to theautomatic response AI, and then, the function image 122 may betransmitted to the automatic response AI.

As described above, when the function images 122 and 126 are transmittedto the automatic response AI, and the password setting function and thedata transmitting function are identified as functions to be linked, thecontroller 26 displays the information of the functions identified asfunctions to be linked, in the display area 48 a as a message content128 of the automatic response AI. When the user approves the messagecontent 128 (for example, when the user makes a message content 130answering “Yes”), information on linkage functions executable by usingthe password setting function and the data transmitting function isdisplayed in the display area 48 a.

The specifying unit 28 specifies the linkage function associated withthe combination of the password setting function and the datatransmitting function (for example, the “function of setting passwordfor data and transmitting the data”) in the linkage function managementtable illustrated in FIG. 26, at a timing when the function images 122and 126 are transmitted as information of functions to be linked to theautomatic response AI, or at a timing when the user approves the messagecontent 128. Accordingly, the linkage functions executable by using thepassword setting function and the data transmitting function arespecified. The information on the linkage functions specified asdescribed above is displayed in the display area 48 a as a messagecontent of the automatic response AI. In addition, the process ofspecifying the functions or the linkage functions may be performed bythe server 16.

FIG. 28 illustrates the screen 48 which is continuous to FIG. 27. Whenthe linkage functions are specified as described above, the controller26 of the terminal device 10 displays the information on the linkagefunctions as a message content 132 of the automatic response AI on thescreen 48.

When the user designates a linkage function and gives an instruction toexecute the linkage function by a message content 134 in response to themessage content 132, the controller 26 receives the executioninstruction and executes the linkage function. In addition, thecontroller 26 displays information indicating that the executioninstruction has been received, as a message content 136 of the automaticresponse AI on the screen 48. As another example, the screen 48 maydisplay a button image for instructing execution of a linkage function,and the user may press the button image to make the instruction toexecute the linkage function. When the password setting function and thedata transmitting function are selected as functions to be linked, thecontroller 26 activates the password setting software to set a passwordfor data to be processed, and further, activates the data transmittingsoftware to transmit the data to a reception destination.

In addition, the function images displayed in the stock area 124 in FIG.27 or the information on the functions associated with the functionimages may be edited by the user. For example, when the user designatesa function image displayed in the stock area 124 and gives aninstruction to edit the function image, the controller 26 displays anediting screen on the display unit of the UI unit 20. For example, thecontroller 26 displays on the display unit of the UI unit 20 the editingscreen when the user presses the function image for a long time (forexample, the length of time during which the function image is pressedbecomes equal to or longer than a threshold), when the user presses abutton image for displaying the editing screen, or when the user inputsan editing instruction to the terminal device 10 by voice. On theediting screen, the function image may be edited, and for example, whena function is assigned to each portion of the function image, the imageof the portion may be cut out from the function image. In addition, theinformation on the function associated with the function image may alsobe edited.

As another example, a function to be linked may be preset as a basiclinkage function. The information of the basic linkage function may bestored in advance in the memory 24 of the terminal device 10. The usermay designate the basic linkage function by using the UI unit 20. Whenthe basic linkage function is set, the user transmits a function imageassociated with a function to be linked other than the basic linkagefunction, as a message content to the automatic response AI.Accordingly, the function to be linked is specified (identified), and alinkage function executable by using the basic function to be linked andthe specified function is specified (identified).

As described above, according to the second exemplary embodiment,information on a linkage function is provided when information onfunctions to be linked is transmitted as a conversation content to theautomatic response AI, on the screen 48 for making a conversation withthe automatic response AI. Thus, the information on a linkage functionis provided by the simple method using a conversation with the automaticresponse AI.

In addition, when three or more function images are transmitted as aconversation content to the automatic response AI, information on alinkage function executable by using the three or more function isprovided.

In addition, when information of plural functions is transmitted as amessage content of the user to the automatic response AI within apredetermined time limit, the specifying unit 28 may specify a linkagefunction executable by using the plural functions, and when informationof one function is transmitted as a message content of the user to theautomatic response AI within the time limit, the specifying unit 28 mayspecify the function as a solo function. In this case, the informationon the linkage function or the solo function specified by the specifyingunit 28 is displayed in the display area 48 a as a message content ofthe automatic response AI. A starting time point of the time limit maybe, for example, a time point when the automatic response AI requeststhe user to inform functions to be linked as in the message content 120,or a time point when the user transmits a first function image to theautomatic response AI after the message content 120 occurs.

For example, in the example illustrated in FIG. 27, in a case where theuser transmits only the function image 122 as a message content of theuser to the automatic response AI within the time limit from the timepoint when the automatic response AI requests the user to informfunctions to be linked as in the message content 120, the specifyingunit 28 identifies the function (the solo function) associated with thefunction image 122. The controller 26 displays the information on thesolo function in the display area 48 a as a message content of theautomatic response AI.

Meanwhile, in a case where the user transmits the function images 122and 126 as a message content of the user to the automatic response AIwithin the time limit from the time point when the automatic response AIrequests the user to inform functions to be linked in the messagecontent 120, the specifying unit 28 specifies linkage functionsassociated with the combination of the password setting function and thedata transmitting function in the linkage function management tableillustrated in FIG. 26. The controller 26 displays the information onthe linkage functions in the display area 48 a.

The process described above may be identically performed in a case wherethe starting time point of the time limit is the time point when theuser transmits a first function image (for example, the function image122) to the automatic response AI.

By providing the time limit as described above, a switching betweenproviding a linkage function and providing a solo function is possible.

In addition, when a linkage function associated with the combination ofthe plural functions designated by the user is not registered in thelinkage function management table, the specifying unit 28 determinesthat the usable linkage function is not registered. In this case, thecontroller 26 displays a message indicating that the usable linkagefunction is not registered, in the display area 48 a as a messagecontent of the automatic response AI. Even in this case, the linkagefunction may become usable according to, for example, an updating statusof a function. In this case, the specifying unit 28 specifies thelinkage function that becomes usable.

In addition, Modifications 1 to 9 of the first exemplary embodiment maybe also applied to the second exemplary embodiment.

FIG. 29 illustrates a process in a case where Modification 1 is appliedto the second exemplary embodiment. As Modification 1 is applied, wheninformation of plural functions to be linked is transmitted as a messagecontent of the user to the automatic response AI, a linkage functionexecutable by using the plural functions is notified. Hereinafter, theprocess in a case where Modification 1 is applied to the secondexemplary embodiment will be described with referent to FIG. 29.

As in FIG. 27, the message content that “Which function do you want touse?” is displayed in the display area 48 a as a message content 138 ofthe automatic response AI.

When the user designates an image 140 representing plural functions andtransmits the image 140 as a message content of the user to theautomatic response AI in response to the message content 138 above, thespecifying unit 28 identifies the plural functions represented in theimage 140 as functions to be linked. For example, when the image 140represents the function image 122 associated with the password settingfunction and the function image 126 associated with the datatransmitting function, the specifying unit 28 identifies the passwordsetting function as a function to be linked based on the function image122, and identifies the data transmitting function as a function to belinked based on the function image 126.

The user may select the function images 122 and 126 from the stock area124 and superimpose the function images 122 and 126 on the image 140, byoperating the screen 48. Accordingly, the function images 122 and 126are represented in the image 140, and the image 140 is transmitted as amessage content of the user to the automatic response AI.

As described above, when the password setting function and the datatransmitting function are identified as described above, the controller26 displays the information of the functions identified as functions tobe linked, in the display area 48 a as a message content 142 of theautomatic response AI. When the user approves the message content 142(for example, when the user makes a message content 144 answering“Yes”), information on the linkage functions executable by using thepassword setting function and the data transmitting function isdisplayed as a message content 146 of the automatic response AI on thescreen 48. When the user designates a linkage function by a messagecontent 148 and gives an instruction to execute the linkage function inresponse to the message content 146, the controller 26 activates thepassword setting software and the data transmitting software.Accordingly, the linkage function is executed by the password settingsoftware and the data transmitting software.

In addition, when three or more function images are represented in theimage 140, information on a linkage function executable by using thethree or more functions is notified. In addition, when one functionimage is represented in the image 140, information on a function (a solofunction) associated with the function is notified.

In addition, when a function image associated with a function to belinked is selected from the stock area 124, the information on a linkagefunction or a solo function may be notified under the control using thetime limit described above. For example, when plural function images aretaken out from the stock area 124 and superimposed on the image 140within the time limit, the specifying unit 28 specifies a linkagefunction executable by using the functions associated with therespective plural function images. In this case, the information on thelinkage function is displayed in the display area 48 a as a messagecontent of the automatic response AI. Meanwhile, when one function imageis taken out from the stock area 124 and superimposed on the image 140within the time limit, the specifying unit 28 specifies a function (asolo function) associated with the function image. In this case, theinformation on the solo function is displayed in the display area 48 aas a message content of the automatic response AI.

For example, in the example illustrated in FIG. 29, when the userselects the function image 122 associated with the password settingfunction from the stock area 124 and superimposes the function image 122on the image 140 without superimposing another function image on theimage 140 within the time limit, the specifying unit 28 specifies thepassword setting function associated with the function image 122 as asolo function. The controller 26 displays the information on the solofunction in the display area 48 a as a message content of the automaticresponse AI.

Meanwhile, when the user selects the function images 122 and 126 fromthe stock area 124 and superimposes the function images 122 and 126 onthe image 140 within the time limit, the specifying unit 28 specifieslinkage functions associated with the combination of the passwordsetting function and the data transmitting function in the linkagefunction management table illustrated in FIG. 26. The controller 26displays the information on the linkage functions in the display area 48a as a message content of the automatic response AI.

In addition, as Modification 2 is applied, when plural function imagesare transmitted as a message content of the user to the automaticresponse AI, a composite image where the plural function images aresuperimposed on each other (see FIG. 14A), or a composite image wherethe plural function images are adjacent and connected to each other (seeFIG. 14B) may be generated. The composite image may be displayed in thestock area 124. In addition, the composite image may be registered as ashortcut image in the terminal device 10.

In addition, as Modification 3 is applied, a linkage function to benotified may be changed according to an order of transmittinginformation of a function to the automatic response AI. In this case, inthe linkage function management table according to the second exemplaryembodiment as well, an access order and a priority of a function imageare set as in the linkage function management table illustrated in FIG.15.

In addition, as Modification 4 is applied, different functions may beassigned to respective portions in a function image. In this case, whenportion images associated with functions are transmitted as a messagecontent of the user to the automatic response AI, information on alinkage function executable by using the functions associated with theportion images is displayed. In addition, a combination of Modifications3 and 4 may be applied to the second exemplary embodiment.

In addition, as Modification 5 is applied, information on a linkagefunction may be first input as a message content of the user by the userto the terminal device 10, and then, information on functions to belinked may be input as a message content of the user to the terminaldevice 10.

In addition, as Modification 6 is applied, the notification of a linkagefunction may be controlled according to information obtained by theautomatic response AI. For example, the automatic response AI mayacquire information indicating a use history of a function, informationindicating an execution status of software for implementing a function,information on the user (information indicating an operation history ora behavior history), and information indicating the user's schedule, bythe learning function, and may control the notification of a linkagefunction by using the pieces of information.

In addition, as Modification 7 is applied, a function to be linked maybe recommended. For example, when the user transmits a function image asa message content of the user to the automatic response AI, thespecifying unit 28 identifies the function associated with the functionimage and specifies a function capable of executing a linkage functionin combination with the identified function by referring to the linkagefunction management table. In addition, the specifying unit 28 mayrecommend a function based on a use history of a linkage function, ormay recommend a function having a relatively high use frequency.

In addition, as Modification 8 is applied, the automatic response AI mayrecommend a function to be used for a linkage function based on theinformation shared with another automatic response AI.

In addition, as Modification 9 is applied, information on functions tobe linked may be input to the terminal device 10 by voice input.

Third Exemplary Embodiment

Hereinafter, a device system according to a third exemplary embodimentwill be described. The third exemplary embodiment corresponds to acombination of the first and second exemplary embodiments. That is, wheninformation of a device and a function (for example, a device image anda function image) is transmitted as a message content of the user to theautomatic response AI, a linkage function executable by using the deviceand the function is notified. Hereinafter, the third exemplaryembodiment will be described in detail.

FIG. 30 illustrates an example of a linkage function management tableaccording to the third exemplary embodiment. The linkage functionmanagement table is information indicating a linkage function executableby using a device and a function (for example, software). In the linkagefunction management table, for example, a combination of IDs (acombination of a device ID and a function ID), information indicating adevice name and a function name, and a linkage function executable byusing a device and a function are associated with each other. A linkagefunction executable by using a device and a function is specified byreferring to the linkage function management table. For example, alinkage function of transmitting data by using the PC A is implementedby combining the PC A which is a device A and the data transmittingfunction which is a function a. Similarly, executable linkage functionsare also set for other devices and other functions. In addition, alinkage function executable by using a total of three or more devicesand functions may be set. In this case, the combination of the totalthree or more devices and functions and a linkage function areassociated with each other, and the association is registered in thelinkage function management table.

Hereinafter, the third exemplary embodiment will be described by usingspecific examples.

FIG. 31 illustrates the screen 48. As in FIG. 8, the screen 48 displaysthe image 50 associated with the automatic response AI, the image 52associated with the user, and information indicating a conversationbetween the user and the automatic response AI.

For example, a message content 150 that “Which function do you want touse?” is displayed in the display area 48 a as a message content of theautomatic response AI.

When the user designates a device in response to the message content150, the specifying unit 28 (the specifying unit 28 implemented by theautomatic response AI) identifies the device as a function to be linked.For example, as illustrated in FIG. 31, when the user designates thedevice image 60 associated with the PC A which is a device, and placesthe device image 60 at the position where a message content of the useris displayed, the controller 26 transmits the device image 60 as amessage content of the user to the automatic response AI. In response tothis operation, the specifying unit 28 identifies the PC A associatedwith the device image 60 as a device to be linked. In the exampleillustrated in FIG. 31, the operation by the user to display the deviceimage 60 in the display area 48 b corresponds to an act for the user tomake a message content. In addition, the user may input a characterstring indicating a device as a message content of the user to theterminal device 10 by operating the UI unit 20. In this case, thespecifying unit 28 identifies the device indicated by the characterstring as a device to be linked. In addition, information of a device tobe linked may be input by voice as a message content of the user to theterminal device 10.

In addition, as in the first exemplary embodiment, the controller 26 maydisplay the stock area 58 the display area 48 b for the user, anddisplay device images in the stock area 58. In this case, the user mayselect a device image in the stock area 58 and move the device image tothe position where a message content of the user is displayed. In thiscase, the controller 26 transmits the device image as a message contentof the user to the automatic response AI.

In the example illustrated in FIG. 31, in addition to the device image60, the user designates a function after a message content 152 of theautomatic response AI. In this case, the specifying unit 28 identifiesthe function as a function to be linked. For example, when the userdesignates a function image 154 associated with a function and placesthe function image 154 at the position where a message content of theuser is displayed, the controller 26 transmits the function image 154 asa message content of the user to the automatic response AI. In responseto this operation, the specifying unit 28 identifies the functionassociated with the function image 154 as a function to be linked. Inthe example illustrated in FIG. 31, the operation by the user to displaythe function image 154 in the display area 48 b corresponds to an actfor the user to make a message content. In addition, the user may inputa character string indicating a function as a message content of theuser to the terminal device 10 by operating the UI unit 20. In thiscase, the specifying unit 28 identifies the function indicated by thecharacter string as a function to be linked. In addition, information ofa function to be linked may be input by voice as a message content ofthe user to the terminal device 10.

The function image 154 is an image associated with an image formingfunction (for example, an image associated with image forming softwareor an image associated with a function of a multifunction machine). Whenthe user selects the function image 154 and makes an instruction toactivate the image forming function, for example, the image formingsoftware is activated, and a printing instruction or a scanninginstruction is assigned to the multifunction machine.

When the user transmits the information of the device and the functionto the automatic response AI on the screen 48, that is, when the usermakes a message indicating the information of the device and thefunction as information of objects to be linked, the specifying unit 28specifies linkage functions associated with the combination of thedevice and the function in the linkage function management tableillustrated in FIG. 30. The controller 26 displays the information onthe linkage functions in the display area 48 a as a message content ofthe automatic response AI.

In the example described above, the device image 60 and the functionimage 154 are transmitted as objects to be linked to the automaticresponse AI. In this case, the specifying unit 28 (the specifying unit28 implemented by the automatic response AI) identifies the PC Aassociated with the device image 60 and the image forming functionassociated with the function image 154 as objects to be linked. Inaddition, the device image 60 may be first transmitted to the automaticresponse AI, and then, the function image 154 may be transmitted to theautomatic response AI, or the function image 154 may be firsttransmitted to the automatic response AI, and then, the device image 60may be transmitted to the automatic response AI.

As described above, when the device image 60 and the function image 154are transmitted to the automatic response AI, and the device and thefunction to be linked are identified, the controller 26 may display theinformation of the device and the function identified as objects to belinked, in the display area 48 a as a message content of the automaticresponse AI. When the user approves the message content (for example,when the user makes a message content answering “Yes”), the informationon the linkage functions executable by using the PC A and the imageforming function may be displayed in the display area 48 a.

The specifying unit 28 specifies the linkage functions associated withthe combination of the PC A associated with the device image 60 and theimage forming function associated with the function image 154 (forexample, the “scanning transmitting function” and the “printingfunction”). Accordingly, the linkage functions executable by using thePC A and the image forming function are specified. The information onthe linkage functions specified as described above is displayed in thedisplay area 48 a as a message content of the automatic response AI. Inaddition, the process of specifying the functions or the linkagefunctions may be performed by the server 16.

The controller 26 displays the information on the linkage functions inthe display area 48 a as a message content 156 of the automatic responseAI.

When the user designates a linkage function and gives an instruction toexecute the linkage function by a message content in response to themessage content 156, the controller 26 receives the executioninstruction and executes the linkage function. For example, when aninstruction to execute the scanning transmitting function is given, amultifunction machine is designated by the user, and data generated byscanning by the multifunction machine is transmitted from themultifunction machine to the PC A, by using, for example, the imageforming software.

As another example, for example, when a device image associated with themultifunction machine B and a function image associated with the datatransmitting function are transmitted as a message content of the userto the automatic response AI, the specifying unit 28 identifies themultifunction machine B and the data transmitting function as objects tobe linked. In this case, the specifying unit 28 specifies linkagefunctions executable by using the multifunction machine B and the datatransmitting software, by referring to the linkage function managementtable illustrated in FIG. 30. The information indicating the linkagefunctions is displayed in the display area 48 a as a message content ofthe automatic response AI.

As yet another example, a device to be linked may be set as a basiclinkage device in advance, or a function to be linked may be set as abasic linkage function in advance. The information of the basic linkagedevice or the basic linkage function may be stored in advance in thememory 24 of the terminal device 10. The user may designate the basiclinkage device or the basic linkage function by using the UI unit 20.When the basic linkage device or the basic linkage function is set, theuser transmits information of a device or a function to be linked otherthan the basic linkage device or the basic linkage function as a messagecontent of the user to the automatic response AI. Accordingly, thedevice or function to be linked is specified (identified).

As described above, according to the third exemplary embodiment,information on a linkage function is provided when information of adevice and a function to be linked is transmitted as a conversationcontent to the automatic response AI, on the screen 48 for making aconversation with the automatic response AI. Thus, the information on alinkage function is provided by the simple method using a conversationwith the automatic response AI.

When a total of three or more device images and function images aretransmitted as a message content of the user to the automatic responseAI, information indicating a linkage function executable by using thethree or more devices and functions may be displayed as a messagecontent of the automatic response AI.

In addition, when information of a device and a function is transmittedas a message content of the user to the automatic response AI within apredetermined time limit, the specifying unit 28 may specify a linkagefunction executable by using the device and the function, and wheninformation of one device or one function is transmitted as a messagecontent of the user to the automatic response AI within the time limit,the specifying unit 28 may specify the function that the one device hasor the one function as a solo function. In this case, information on thelinkage function or the solo function specified by the specifying unit28 is displayed on the display unit of the UI unit 20. A starting timepoint of the time limit may be, for example, a time point when theautomatic response AI requests the user to inform objects to be linkedas in the message content 150, or a time point when the user transmits afirst device image or a first function image to the automatic responseAI after the message content 150 occurs.

For example, in the example illustrated in FIG. 31, in a case where theuser transmits only the device image 60 as a message content of the userto the automatic response AI within the time limit from the time pointwhen the automatic response AI requests the user to inform objects to belinked as in the message content 150, the specifying unit 28 identifiesa function (a solo function) that the device associated with the deviceimage 60 has. The controller 26 displays the information on the solofunction in the display area 48 a as a message content of the automaticresponse AI.

Meanwhile, in a case where the user transmits the device image 60 andthe function image 154 as a message content of the user to the automaticresponse AI within the time limit from the time point when the automaticresponse AI requests the user to inform objects to be linked as in themessage content 150, the specifying unit 28 specifies linkage functionsassociated with the combination of the PC A and the image formingfunction in the linkage function management table illustrated in FIG.30. The controller 26 displays the information on the linkage functionsin the display area 48 a.

The process described above may be identically performed in a case wherethe starting time point of the time limit is the time point when theuser transmits a first image (for example, the device image 60) to theautomatic response AI.

By providing the time limit as described above, a switching betweenproviding a linkage function and providing a solo function is possible.

In addition, when a linkage function associated with the combination ofthe device and the function designated by the user is not registered inthe linkage function management table, the specifying unit 28 determinesthat the usable linkage function is not registered. In this case, thecontroller 26 displays a message indicating that the usable linkagefunction is not registered, in the display area 48 a as a messagecontent of the automatic response AI. Even in this case, the linkagefunction may become usable according to, for example, an updating statusof a device or a function. In this case, the specifying unit 28specifies the linkage function that becomes usable.

In addition, Modifications 1 to 9 of the first exemplary embodiment mayalso be applied to the third exemplary embodiment.

FIG. 32 illustrates a process in a case where Modification 1 is appliedto the third exemplary embodiment. As Modification 1 is applied, wheninformation of a device and a function which are objects to be linked istransmitted as a message content of the user to the automatic responseAI, a linkage function executable by using the device and the functionis notified. Hereinafter, the process in a case where Modification 1 isapplied to the third exemplary embodiment will be described withreference to FIG. 32.

As in FIG. 31, the message content that “Which function do you want touse?” is displayed in the display area 48 a as a message content 158 ofthe automatic response AI.

When the user designates an image 160 representing a device and afunction and transmits the image 160 as a message content of the user tothe automatic response AI in response to the message content 158 above,the specifying unit 28 identifies the device and the function which arerepresented in the image 160 as objects to be linked. For example, whenthe image 160 represents the device image 60 and the function image 154,the specifying unit 28 identifies the PC A as a device to be linkedbased on the device image 60, and identifies the image forming functionas a function to be linked based on the function image 154.

The user may select the device image 60 from the stock area 58 andsuperimpose the device image 60 on the image 160, by operating thescreen 48. Similarly, the user may select the function image 154 fromthe stock area 124 and superimpose the function image 154 on the image160, by operating the screen 48. Accordingly, the device image 60 andthe function image 154 are represented in the image 160, and the image160 is transmitted as a message content of the user to the automaticresponse AI.

As described above, when the PC A and the image forming function areidentified as objects to be linked, the controller 26 may display theinformation of the device and the function which are identified asobjects to be linked, in the display area 48 a as a message content ofthe automatic response AI. When the user approves the message content(for example, when the user makes a message content answering “Yes”),information on linkage functions executable by using the PC A and theimage forming function may be displayed in the display area 48 a as amessage content 162 of the automatic response AI. When the userdesignates a linkage function and gives an instruction to execute thelinkage function by a message content 164 in response to the messagecontent 162, the controller 26 executes the designated linkage function.

In addition, when the image 160 represents a total of three of moredevice images and function images, information on a linkage functionexecutable by using the total three or more devices and functions isnotified. In addition, when the image 160 represents one device image orone function image, information on a function (a solo function) that afunction associated with the one device image has or a function (a solofunction) associated with the one function image is notified.

In addition, when a device image or a function image is selected fromthe stock area, the information on a linkage function or a solo functionmay be notified under the control using the time limit described above.For example, when a device image and a function image are taken out fromthe stock area and superimposed on the image 160 within the time limit,the specifying unit 28 specifies a linkage function executable by usingthe device associated with the device image and the function associatedwith the function image. In this case, the information on the linkagefunction is displayed in the display area 48 a as a message content ofthe automatic response AI. Meanwhile, when one device image or onefunction image is taken out from the stock area and superimposed on theimage 160 within the time limit, the specifying unit 28 specifies afunction (a solo function) that the device associated with the deviceimage has or a function (a solo function) associated with the functionimage. In this case, the information on the solo function is displayedin the display area 48 a as a message content of the automatic responseAI.

In addition, as Modification 2 is applied, when a device image and afunction image are transmitted as a message content of the user to theautomatic response AI, a composite image where the device image and thefunction image are superimposed on each other (see FIG. 14A), or acomposite image where the device image and the function image areadjacent and connected to each other (see FIG. 14B) may be generated.The composite image may be displayed in the stock area. In addition, thecomposite image may be registered as a shortcut image in the terminaldevice 10.

In addition, as Modification 3 is applied, a linkage function to benotified may be changed according to an order of transmittinginformation of a device and a function to the automatic response AI. Inthis case, in the linkage function management table according to thethird exemplary embodiment as well, access orders and priorities of adevice image and a function image are set as in the linkage functionmanagement table illustrated in FIG. 15.

In addition, as Modification 4 is applied, different functions may beassigned to respective portions in a device image, or differentfunctions may be assigned to respective portions in a function image. Inthis case, when portion images associated with portions of a device aretransmitted as a message content of the user to the automatic responseAI, information on a linkage function executable by using the portionsis displayed. Similarly, when portion images associated with functionsare transmitted as a message content of the user to the automaticresponse AI, information on a linkage function executable by using thefunctions associated with the portion images is displayed. In addition,a combination of Modifications 3 and 4 may be applied to the thirdexemplary embodiment.

In addition, as Modification 5 is applied, information on a linkagefunction may be first input as a message content of the user by the userto the terminal device 10, and then, information on a device and afunction to be linked may be input as a message content of the user tothe terminal device 10.

In addition, as Modification 6 is applied, the notification of a linkagefunction may be controlled according to information obtained by theautomatic response AI. For example, the automatic response AI mayacquire information indicating a use history of a device and a function,information indicating an execution status of a device, informationindicating an execution status of software for implementing a function,information on the user (information indicating an operation history ora behavior history), and information indicating the user's schedule, bythe learning function, and may control the notification of a linkagefunction by using the pieces of information.

In addition, as Modification 7 is applied, a function to be linked maybe recommended. For example, when the user transmits a device image as amessage content of the user to the automatic response AI, the specifyingunit 28 identifies the device associated with the device image, andspecifies a function capable of executing a linkage function incombination with the device, by referring to the linkage functionmanagement table. The function is recommended to the user. Similarly, adevice to be linked may be recommended. For example, when the usertransmits a function image as a message content of the user to theautomatic response AI, the specifying unit 28 identifies the functionassociated with the function image, and specifies a device capable ofexecuting a linkage function in combination with the function, byreferring to the linkage function management table. The device isrecommended to the user. In addition, the specifying unit 28 mayrecommend a device or a function based on a use history of a linkagefunction, or may recommend a device or a function having a relativelyhigh use frequency.

In addition, as Modification 8 is applied, the automatic response AI mayrecommend a function to be used for a linkage function based on theinformation shared with another automatic response AI.

In addition, as Modification 9 is applied, information on a device and afunction to be linked may be input to the terminal device 10 by voiceinput.

Fourth Exemplary Embodiment

Hereinafter, a device system according to a fourth exemplary embodimentwill be described. In the fourth exemplary embodiment, when informationof files (data) is transmitted as a message content of the user to theautomatic response AI, information on a linkage function is notified asa message content of the automatic response AI. The information of afile is identification information for identifying the file and is, forexample, a file image associated with a file or character information orvoice information indicating the identification information. Forexample, when the user transmits a file image associated with a file tothe automatic response AI, when the user provides the automatic responseAI with a character string indicating identification information of afile by inputting the character string to the terminal device 10, orwhen the user provides the automatic response AI with identificationinformation of a file by voice, the specifying unit 28 identifies thefile. Hereinafter, the fourth exemplary embodiment will be described indetail.

The linkage function management information according to the fourthexemplary embodiment is information for managing a linkage function thatis executed by linking plural functions to each other. The linkagefunction is a function implemented by linking plural pieces of softwareto each other. In addition, the linkage function may be a functionimplemented by linking a function that a hardware device has and afunction implemented by software to each other.

The linkage function according to the fourth exemplary embodiment willbe described in more detail. The linkage function is a function executedon plural files (data) when the user selects the plural files. Forexample, each file is associated with software (application) forexecuting a process on the file. The linkage function is implemented bylinking applications associated with respective plural files selected bythe user to each other.

The files are, for example, a document file, an image file, a videofile, a spreadsheet file, a presentation file, a graphic (drawing) file,and an audio file.

The document file is a file having a document format and is configuredwith character strings or other pieces of information. The document filemay contain images, tables, or figures as other pieces of information.The document format is, for example, a text format or a document format.The document file corresponds to, for example, data having a text format(for example, data appended with a character string “.txt” as anextension) or data having a document format (for example, data appendedwith a character string “.doc” or “.docx” as an extension).

The image file is a file having an image format. The image file may beimage data having a raster format or image data having a vector format.The image format is, for example, a JPEG format, a BMP (bitmap) format,a PDF format, a GIF format, a TIFF format, or a PICT format. The imagefile corresponds to, for example, data having a JPEG format (dataappended with a character string “.jpeg” as an extension), data having aBMP format (data appended with a character string “.bmp” as anextension), data having a PDF format (data appended with a characterstring “.pdf” as an extension), data having a GIF format (data appendedwith a character string “.gif” as an extension), data having a TIFFformat (data appended with a character string “.tiff” as an extension),data having a PNG format (data appended with a character string “.png”as an extension), or data having a PICT format (data appended with acharacter string “.pict” as an extension). In addition, image datahaving a format other than the formats described above may be used asthe image file according to the fourth exemplary embodiment.

The video file is a file having a video format. The video format is, forexample, an AVI format, an MP4 format, an MOV format, an MPEG2-TSformat, an MPEG2-PS format, a WMV format, or an FLV format. The videofile corresponds to, for example, data having an AVI format (dataappended with a character string “.avi” as an extension), data having anMP4 format (data appended with a character string “.mp4” as anextension), data having an MOV format (data appended with a characterstring “.mov” as an extension), data having an MPEG2-TS format (dataappended with a character string “.m2ts” as an extension), data havingan MPEG2-PS format (data appended with a character string “.mpeg” as anextension), data having a WMV format (data appended with a characterstring “.wmv” as an extension), or data having an FLV format (dataappended with a character string “.flv”). In addition, video data havinga format other than the formats described above may be used as the videofile according to the fourth exemplary embodiment.

The spreadsheet file is a file having a table format and is configuredwith tables, graphs, or other pieces of information. The spreadsheetfile may contain character strings, images, or figures as other piecesof information. For example, the spreadsheet file is data appended witha character string “.csv,” “.xls” or “.xlsx” as an extension.

The presentation file is a file having a presentation sheet format. Forexample, the presentation file corresponds to data appended with acharacter string “.ppt” or “.pptx” as an extension. The presentationfile is configured with, for example, character strings, images (stillimages or video), figures, and voice.

The graphic file is a file having a graphic (drawing) format. Thegraphic file is created by, for example, graphic generating software(for example, two-dimensional CAD or three-dimensional CAD). Forexample, the graphic file corresponds to data having a DXF format (dataappended with a character string “.dxf” as an extension) or data havinga DWG format (data appended with a character string “.dwg” as anextension). In addition, data appended with a character string “.vsd” or“.vsdx” as an extension may be treated as the graphic file.

The audio file is a file having an audio format. The audio format is,for example, an uncompressed audio format (for example, a WAV format, anAIFF format, or a BWF format), an irreversible compressed audio format(for example, an mp3 format, an AAC format, or a WMA format), or areversible compressed audio format (for example, a TAK format or an FLACformat). For example, the audio file corresponds to data having anuncompressed audio format (data appended with, for example, a characterstring “.wav” as an extension), data having an irreversible compressedaudio format (data appended with, for example, a character string “.mp3”as extension), or data having an uncompressed audio format (dataappended with, for example, a character string “.tak” as an extension).

In the fourth exemplary embodiment, a linkage function is set for eachfile format (attribute), and the linkage function management informationis, for example, information indicating an association betweeninformation indicating a combination of plural file formats andinformation indicating a linkage function (linkage functioninformation). The linkage function information is, for example, alinkage function ID or a linkage function name. The file format is, forexample, the above-described document format, image format, videoformat, table format, sheet format, graphic format, or audio format. Alinkage function associated with the combination of the plural fileformats is specified (identified) by referring to the linkage functionmanagement information.

The linkage function may be a function implemented by linking pluraldifferent functions (for example, different pieces of software) to eachother or a function implemented by linking identical functions (forexample, identical pieces of software) to each other. The linkagefunction may be a function which is unusable before the linkage. Afunction which is unusable before linkage may become usable by linkingidentical functions to each other, or become usable by combiningdifferent functions to each other. For example, when a document creatingfunction (document creating software) and an image displaying function(image displaying software (for example, an image viewer)) are linked toeach other, a function of inserting (attaching) an image into a documentor a function of superimposing a document (a character string or a text)on an image is implemented as a linkage function. That is, when thedocument creating function and the image displaying function are linkedto each other, an image inserting function or a text superimposingfunction is implemented.

The concept of the linkage function may include a coalescing functionexecutable as a new function by combining plural functions with eachother. For example, an expanded displaying function may be implementedas the coalescing function by combining plural displaying functions. Inaddition, a capturing area expanding function may be implemented as thecoalescing function by combining plural capturing functions. Inaddition, a translated call function (a function of translating a phoneconversation) may be implemented as the coalescing function by combininga call function and a translation function. As described above, theconcept of the linkage function may include a function which may beimplemented by linking identical types of functions to each other, or afunction which may be implemented by linking different types offunctions to each other.

A linkage function managed by the linkage function managementinformation may be a function executable by pieces of software installedin the terminal device 10, a function executable by pieces of softwareinstalled in an external device (for example, the server or anotherterminal device), or a function executable by software installed in theterminal device 10 and software installed in an external device.

In addition, as another example of the linkage function managementinformation, information indicating a combination of plural functions(plural pieces of software), and information indicating a linkagefunction executable by linking the plural functions to each other(linkage function information) may be associated with each other.

When file images associated with plural files, respectively, aretransmitted as a message content of the user to the automatic responseAI, the specifying unit 28 specifies a linkage function associated witha combination of the formats of the plural files (file formats) in thelinkage function management information. Accordingly, the executablelinkage function is specified.

Hereinafter, the linkage function management information according tothe fourth exemplary embodiment will be described in detail withreference to FIG. 33. FIG. 33 illustrates an example of a linkagefunction management table according to the fourth exemplary embodiment.In the linkage function management table, for example, informationindicating a combination of plural file formats and informationindicating a linkage function (linkage function information) areassociated with each other. For example, when the user transmitsinformation of plural files as a message content of the user to theautomatic response AI, the specifying unit 28 specifies a linkagefunction associated with the combination of the file formats of theplural files in the linkage function management table. The informationon the linkage function is displayed on the display unit of the UI unit20. In addition, while FIG. 33 illustrates a combination of two fileformats, an association between a combination of three or more fileformats and a linkage function may be registered in the linkage functionmanagement table.

Hereinafter, each linkage function illustrated in FIG. 33 will bedescribed.

In FIG. 33, the “document format” is a text format (with an extension of“.txt”) or a document format (with an extension of “.doc” or “.docx”).The “table format” is a format with an extension expressed as, forexample, “.csv” or “.xls.” The “image format” is a format with anextension expressed as, for example, “.jpeg” or “.pdf” The “videoformat” is a format with an extension expressed as, for example, “.avi”or “.mpeg.” The “sheet format” is a format with an extension expressedas, for example, “.ppt.” While FIG. 33 does not illustrate the graphicformat or the audio format, an association between a combination of fileformats including the graphic format and the audio format and a linkagefunction may be registered in the linkage function management table. Inaddition, an association between a combination of file formats otherthan the file formats illustrated in FIG. 33 and a linkage function maybe set and registered in the linkage function management table.

Hereinafter, each linkage function illustrated in FIG. 33 will bedescribed. When the user transmits information of a document file A anda spreadsheet file B as a message content of the user to the automaticresponse AI, a “function of inserting a table into a document” and a“function of inserting a document into a table” which are associatedwith the combination of the document format and the table format arespecified as linkage functions. The “function of inserting a table intoa document” is a function of inserting the table represented by thespreadsheet file B into the document represented by the document file A.The “function of inserting a document into a table” is a function ofinserting the document represented by the document file A into the tablerepresented by the spreadsheet file B.

When the user transmits information of the document file A and an imagefile C as a message content of the user to the automatic response AI, a“function of inserting an image into a document” and a “function ofsuperimposing a document on an image” which are associated with thecombination of the document format and the image format are specified aslinkage functions. The “function of inserting an image into a document”is a function of inserting the image represented by the image file Cinto the document represented by the document file A. The “function ofsuperimposing a document on an image” is a function of superimposing thedocument represented by the document file A on the image represented bythe image file C.

When the user transmits information of the document file A and a videofile D as a message content of the user to the automatic response AI, a“function of inserting a video into a document” and a “function ofinserting a still image into a document” which are associated with thecombination of the document format and the video format are specified aslinkage functions. The “function of inserting a video into a document”is a function of inserting the video represented by the video file Dinto the document represented by the document file A. The “function ofinserting a still image into a document” is a function of inserting oneor more frames (still images) included in the video file D into thedocument represented by the document file A. For example, when thedocument file A is a document file which is not compatible with a video,that is, a file into which a video cannot be inserted and when the“function of inserting a still image into a document” is executed, oneor more frames (still images) included in the video file D are insertedinto the document represented by the document file A.

Plural files having the same file format may be selected by the user.For example, when the user transmits information of document files A1and A2 as a message content of the user to the automatic response AI, a“function of integrating documents” which is associated with thecombination of the document formats is specified as a linkage function.The “function of integrating documents” is a function of integrating thedocuments represented by the document files A1 and A2 with each other(for example, combining the two documents with each other), andgenerates a new document file A3 while saving or removing the documentfiles A1 and A2.

When the user transmits information of spreadsheet files B1 and B2 as amessage content of the user to the automatic response AI, a “function ofintegrating tables” which is associated with the combination of thetable formats is specified as a linkage function. The “function ofintegrating tables” is a function of integrating the tables representedby the spreadsheet files B1 and B2 with each other (for example,combining the two tables with each other), and generates a newspreadsheet file B3 while saving or removing the spreadsheet files B1and B2.

When the user transmits information of video files D1 and D2 as amessage content of the user to the automatic response AI, a “function ofintegrating videos” which is associated with the combination of thevideo formats is specified as a linkage function. The “function ofintegrating videos” is a function of integrating the videos representedby the video files D1 and D2 with each other (for example, combining thetwo videos with each other), and generates a new video file D3 whilesaving or removing the video files D1 and D2.

In addition, when the user transmits information of a presentation fileE and the document file A as a message content of the user to theautomatic response AI, a “function of inserting a document into apresentation sheet” which is associated with the combination of thesheet format and the document format is specified as a linkage function.The “function of inserting a document into a presentation sheet” is afunction of inserting the document represented by the document file Ainto the sheet represented by the presentation file E.

Hereinafter, the fourth exemplary embodiment will be described in detailby using specific examples.

FIG. 34 illustrates the screen 48. As in FIG. 8, the screen 48 displaysthe image 50 associated with the automatic response AI, the image 52associated with the user, and information indicating a conversationbetween the user and the automatic response AI.

For example, a message content 166 that “Which file do you want to use?”is displayed in the display area 48 a as a message content of theautomatic response AI.

When the user designates a file in response to the message content 166,the specifying unit 28 (the specifying unit 28 implemented by theautomatic response AI) identifies the file as a file to be linked. Forexample, as illustrated in FIG. 34, when the user designates a fileimage 168 associated with the document file A and places the file image168 at the position where a message content of the user is displayed,the controller 26 transmits the file image 168 as a message content ofthe user to the automatic response AI. In response to this operation,the specifying unit 28 identifies the document file A associated withthe file image 168 as a file to be linked. In the example illustrated inFIG. 34, the operation by the user to display the file image 168 in thedisplay area 48 b corresponds to an act for the user to make a messagecontent. In addition, the user may input a character string indicating afile name as a message content of the user to the terminal device 10 byoperating the UI unit 20. In this case, the specifying unit 28identifies the file having the file name indicated by the characterstring as a file to be linked. In addition, information of a file to belinked may be input by voice as a message content of the user to theterminal device 10.

For example, as in the first exemplary embodiment, the controller 26displays a stock area 170 in the display area 48 b for the user, anddisplays file images in the stock area 170. For example, the controller26 displays file images associated with files stored in the terminaldevice 10, in the stock area 170. In addition, when there exists anotherfile image that may not be displayed in the stock area 170, the otherfile image is displayed in the stock area 170 in the manner that theuser performs an operation to scroll the file images displayed in thestock area 170.

As in the first exemplary embodiment, the user may select a file imageassociated with a file to be linked (the file image 168 in the exampleillustrated in FIG. 34) from the stock area 170, move the file image tothe outside of the stock area 170, and transmit the file image 168 as amessage content of the user to the automatic response AI. For example,the user moves the file image 168 from the stock area 170 to theposition where a message content of the user is displayed, by performingthe dragging operation, and performs the dropping operation at theposition where the message content is displayed. With the operations,the controller 26 transmits the file image 168 as a message content ofthe user to the automatic response AI.

In the example illustrated in FIG. 34, in addition to the file image168, the user selects a file image 172 associated with the image file Cfrom the stock area 170 and moves the file image 172 to the positionwhere a message content of the user is displayed. With the operation,the file image 172 is transmitted as a message content of the user tothe automatic response AI.

When the user transmits information of the plural files to the automaticresponse AI on the screen 48, that is, when the user makes a messageindicating information of the plural files as information of files to belinked, the specifying unit 28 specifies a linkage function associatedwith the plural files in the linkage function management tableillustrated in FIG. 33. The controller 26 displays the information onthe linkage functions in the display area 48 a as a message content ofthe automatic response AI.

In the example described above, the file images 168 and 172 aretransmitted as functions to be linked to the automatic response AI. Inthis case, the specifying unit 28 (the specifying unit 28 implemented bythe automatic response AI) identifies the document file A associatedwith the file image 168 and the image file C associated with the fileimage 172 as files to be linked. In addition, the file image 168 may befirst transmitted to the automatic response AI, and then, the file image172 may be transmitted to the automatic response AI, or the file image172 may be first transmitted to the automatic response AI, and then, thefile image 168 may be transmitted to the automatic response AI.

When the file images 168 and 172 are transmitted to the automaticresponse AI, and the document file A and the image file C are identifiedas files to be linked as described above, the controller 26 displays theinformation of the files identified as files to be linked, in thedisplay area 48 a as a message content of the automatic response AI.When the user approves the message content (for example, when the usermakes a message content answering “Yes”), information on linkagefunctions executable by using the document file A and the image file Cis displayed in the display area 48 a.

The specifying unit 28 identifies the file format of the document file Abased on the extension of the document file A (the extension appended tothe document file A as, for example, attribute information of thedocument file A). Similarly, the specifying unit 28 identifies the fileformat of the image file C based on the extension of the image file C(the extension appended to the image file C as, for example, attributeinformation of the image file C). In addition, the extension may beincluded in a file name. When the extension of the document file Acorresponds to an extension representing the document format such as“.txt” or “.docx,” the specifying unit 28 identifies that the fileformat of the document file A is the document format. When the extensionof the image file C corresponds to an extension representing the imageformat such as “jpeg,” the specifying unit 28 identifies that the fileformat of the image file C is the image format. Then, the specifyingunit 28 specifies the linkage functions associated with the combinationof the document format and the image format (the “function of insertingan image into a document” and the “function of superimposing a documenton an image”) in the linkage function management table illustrated inFIG. 33. The information on the linkage functions specified as describedabove is displayed in the display area 48 a as a message content of theautomatic response AI. In addition, the process of specifying the filesor the linkage functions may be performed by the server 16.

The controller 26 displays the information on the linkage functions inthe display area 48 a as a message content 174 of the automatic responseAI.

When the user designates a linkage function and gives an instruction toexecute the linkage function by a message content in response to themessage content 174, the controller 26 (the controller 26 implemented bythe automatic response AI) receives the execution instruction andexecutes the linkage function. For example, when the user gives aninstruction to execute the “function of inserting an image into adocument,” the controller 26 inserts the image represented by the imagefile C into the document represented by the document file A.

More specifically, the controller 26 activates a document applicationand an image application. The document application is software (program)having a function of creating, editing, and displaying a file having thedocument format. The image application is software (program) having afunction of creating, editing, and displaying a file having the imageformat. In addition, the document application may be associated inadvance with the document file A, and the image application may beassociated in advance with the image file C. The controller 26 copiesthe image of the image file C by using the function that the imageapplication has, and pastes the image into the document represented bythe document file A by using the function that the document applicationhas. Accordingly, the document into which the image has been inserted isgenerated. In addition, the controller 26 may extract a portion imagerepresenting a specific portion from the image represented by the imagefile C by using the function of the image application, and paste theportion image into the document. In the document, the area where theimage is pasted may be a predetermined area, an area designated by theuser, or an area presumed to describe a character string related to theimage. For example, the controller 26 presumes or searches for acharacter string expressing a feature portion represented in the image,and presumes that, in the document, an area describing a characterstring having the same or similar meaning to the searched characterstring is an area describing the character string related to the image.In addition, the area may be presumed by another method.

Similarly, when the user gives an instruction to execute the “functionof superimposing a document on an image,” all or some parts of thedocument represented by the document file A are copied by the functionthat the document application has, and the copied document issuperimposed on the image represented by the image file C. The portionto be copied from the document may be determined in advance ordesignated by the user. In addition, the portion of the image where thedocument is to be superimposed may be determined in advance, ordesignated by the user.

The controller 26 may generate a new file by executing a linkagefunction. In this case, the plural files themselves selected as objectsto be linked are not edited. For example, when the document file A andthe image file C are selected as files to be linked, and an instructionto execute the “function of inserting an image into a document” isgiven, the controller 26 generates a new document file AC by pasting theimage represented by the image file C into the document represented bythe document file A. In this case, the controller 26 may generate a newfile image (for example, an icon) associated with the document file ACand display the file image on the screen 48. In this case, the documentfile A and the image file C themselves are not edited.

Similarly, when an instruction to execute the “function of superimposinga document on an image” is given, the controller 26 generates a newimage file CA by superimposing all or some parts of the documentrepresented by the document file A on the image represented by the imagefile C. In this case as well, the document file A and the image file Cthemselves are not edited.

As another example, the controller 26 may execute the linkage functionto thereby edit (including, for example, process) the files themselvesselected as objects to be linked, without generating a new file.Referring to the example described above, the controller 26 pastes theimage represented by the image file C into the document represented bythe document file A, and saves the document file in which the pasting ofthe image has been reflected, as the document file A. In this case, thedocument file A is updated.

As yet another example, a file to be linked may be preset as a basicfile to be linked. The information of the basic file to be linked may bestored in advance in the memory 24 of the terminal device 10. The usermay designate the basic file to be linked by using the UI unit 20. Whenthe basic file to be linked is set, the user transmits information of afile to be linked, other than the basic file to be linked, as a messagecontent of the user to the automatic response AI. Accordingly, the fileto be linked is specified (identified).

As described above, according to the fourth exemplary embodiment,information on a linkage function is provided when information of filesto be linked is transmitted as a conversation content to the automaticresponse AI, on the screen 48 for making a conversation with theautomatic response AI. Thus, the information on a linkage function isprovided by the simple method using a conversation with the automaticresponse AI.

In addition, when three or more file images are transmitted as a messagecontent of the user to the automatic response AI, information on alinkage function associated with the combination of the three or morefile formats may be displayed.

In the fourth exemplary embodiment, the display of information on alinkage function, or a linkage function to be executed may be changedaccording to a master-servant relationship of files. For example, a fileof which represented contents themselves are editable corresponds to amaster file, and a file of which contents themselves are not editablecorresponds to a servant file. For example, when a document itselfrepresented in a document file is editable, the document filecorresponds to the master file. In addition, when an image itselfrepresented in an image file is not editable (that is, when contents maybe superimposed on an image, but the image itself is not editable), theimage file corresponds to the servant file. This relationship issimilarly applied to other files. In addition, a document file alsocorresponds to the servant file when the document of the document fileis not editable, and an image file also corresponds to the master filewhen the image of the image file is editable.

The specifying unit 28 may determine whether a file corresponds to themaster file or the servant file, based on a file format (that is, anextension). For example, the specifying unit 28 may determine that afile having the document format, a file having the table format, a filehaving the sheet format, and a file having the graphic format correspondto master files, and a file having the image format, a file having thevideo format, and a file having the audio format correspond to servantfiles. In addition, whether a file corresponds to the master file or theservant file may not be determined by collectively determining the fileformats, and may be determined by individually determining filesselected as objects to be linked. For example, even a file having theimage format may correspond to the master file when the image of thefile is editable.

For example, when the file image 168 associated with the document file Aand the file image 172 associated with the image file C are transmittedas a message content of the user to the automatic response AI, thespecifying unit 28 determines whether the document file A corresponds tothe master file or the servant file, based on the extension associatedwith the document file A, and determines whether the image file Ccorresponds to the master file or the servant file, based on theextension associated with the image file C. It is assumed that theextension of the document file A is, for example, “.docx,” and theextension of the image file C is, for example, “.jpeg.” In this case,since the document file A has the document format and is editable, thespecifying unit 28 determines that the document file A corresponds tothe master file. Further, since the image file C has the image formatand is not editable, the specifying unit 28 determines that the imagefile C corresponds to the servant file. In addition, the specifying unit28 may determine whether the document file A and the image file C areeditable, by individually determining the document file A and the imagefile C, rather than the determination based on the extensions.

Since the document file A corresponds to the master file and the imagefile C corresponds to the servant file, the image represented in theimage file C corresponds to a content to be pasted, and the documentrepresented in the document file A corresponds to a content of a placewhere the image is to be pasted. In this case, the specifying unit 28specifies the “function of inserting an image into a document” as alinkage function. The controller 26 displays the information on the“function of inserting an image into a document” in the display area 48a as a message content of the automatic response AI. When the specifyingunit 28 specifies plural linkage functions, the controller 26 displaysthe information on the plural linkage functions in the display area 48 aas a message content of the automatic response AI.

In addition, the controller 26 may display, in the display area 48 a,information on a linkage function that is specified based on themaster-servant relationship of files from among the plural linkagefunctions registered in the linkage function management tableillustrated in FIG. 33, as a message content of the automatic responseAI in preference to the information on the other linkage functions.Referring to the example where the document file A and the image file Care transmitted, the “function of inserting an image into a document”and the “function of superimposing a document on an image” areregistered as linkage functions associated with the combination of thedocument format and the image format in the linkage function managementtable (see FIG. 33). In this case, the specifying unit 28 specifies the“function of inserting an image into a document” which is specifiedbased on the master-servant relationship of files as a linkage functionhaving a “first priority,” and specifies the “function of superimposinga document on an image” as a linkage function having a “secondpriority.” The controller 26 displays the information on the “functionof inserting an image into a document” in the display area 48 a as amessage content of the automatic response AI in preference to (forexample, at a higher rank) the information of the “function ofsuperimposing a document on an image,” according to the priorities.

In addition, when plural linkage functions are specified based on themaster-servant relationship of files, the controller 26 may determinepriorities of the respective linkage functions based on a history of useof each linkage function by the user, and display the information on thelinkage functions as a message content of the automatic response AI onthe screen 48 according to the priorities of the linkage functions. Forexample, the controller 26 more preferentially displays information on alinkage function in the display area 48 a (for example, at a higherrank) as the use frequency of the linkage information is larger.

In addition, when plural linkage functions are specified based on themaster-servant relationship of files, the controller 26 may determinepriorities of the respective linkage functions based on a relationshipof plural files selected as objects to be linked (that is, arelationship of plural file formats), and display the information on thelinkage functions according to the priorities of the linkage functionsin the display area 48 a as a message content of the automatic responseAI. For example, the controller 26 determines a degree of the likelihoodthat each linkage function included in the plural linkage functionsspecified by the specifying unit 28 is to be used, based on plural fileformats. The controller 26 more preferentially displays information on alinkage function in the display area 48 a (for example, at a higherrank) as the use probability of the linkage function is larger. Forexample, in the combination of the document format and the image format,it is predicted that the use likelihood of the “function of inserting animage into a document” is higher than that of the “function ofsuperimposing a document on an image.” In this case, the controller 26displays the information on the “function of inserting an image into adocument” in the display area 48 a in preference to the “function ofsuperimposing a document on an image”.

In addition, when information of plural files is transmitted as amessage content of the user to the automatic response AI within apredetermined time limit, the specifying unit 28 may specify a linkagefunction executable by using the plural files, and when information ofone file is transmitted as a message content of the user to theautomatic response AI within the time limit, the specifying unit 28 mayspecify a solo function executable by using the one file. In this case,information on the linkage function or the solo function specified bythe specifying unit 28 is displayed on the display unit of the UI unit20. A starting time point of the time limit may be a time point when theautomatic response AI requests the user to inform objects to be linkedas in the message content 166, or a time point when the user transmits afirst file image to the automatic response AI after the message content166 occurs.

For example, in the example illustrated in FIG. 34, when the usertransmits only the file image 168 as a message content of the user tothe automatic response AI within the time limit from the time point whenthe automatic response AI requests the user to inform objects to belinked as in the message content 166, the specifying unit 28 identifiesa function (a solo function) executable by using the file associatedwith the file image 168. The controller 26 displays the information onthe solo function in the display area 48 a as a message content of theautomatic response AI.

Meanwhile, when the user transmits the file images 168 and 172 as amessage content of the user to the automatic response AI within the timelimit from the time point when the automatic response AI requests theuser to inform objects to be linked as in the message content 166, thespecifying unit 28 specifies linkage functions associated with thecombination of the document file (the file having the document format)and the image file (the file having the image format) in the linkagefunction management table illustrated in FIG. 33. The controller 26displays the information on the linkage functions in the display area 48a.

The process described above may be identically performed when thestarting time of the time limit is the time point when the usertransmits a first file image (for example, the file image 168) to theautomatic response AI.

By providing the time limit as described above, a switching betweenproviding a linkage function and providing a solo function is possible.

In addition, when a linkage function associated with the combination ofthe plural file formats designated by the user is not registered in thelinkage function management table, the specifying unit 28 determinesthat the usable linkage function is not registered. In this case, thecontroller 26 displays a message indicating that the usable linkagefunction is not registered, in the display area 48 a as a messagecontent of the automatic response AI. Even in this case, the linkagefunction may become usable according to, for example, an updating statusof a device or software. In this case, the specifying unit 28 specifiesthe linkage function that becomes usable.

In addition, Modifications 1 to 9 of the first exemplary embodiment maybe applied to the fourth exemplary embodiment.

FIG. 35 illustrates a process in a case where Modification 1 is appliedto the fourth exemplary embodiment. As Modification 1 is applied, wheninformation of plural files which are objects to be linked istransmitted as a message content of the user to the automatic responseAI, a linkage function executable by using the plural files is notified.Hereinafter, the process in the case where Modification 1 is applied tothe fourth exemplary embodiment will be described with reference to FIG.35.

As in FIG. 34, the message content that “Which file do you want to use?”is displayed in the display area 48 a as a message content 176 of theautomatic response AI.

When the user designates and transmits an image 178 representing pluralfiles as a message content of the user to the automatic response AI inresponse to the message content 176, the specifying unit 28 identifiesthe plural files represented in the image 178 as files to be linked. Forexample, when the image 178 represents the file images 168 and 172, thespecifying unit 28 identifies the document file A as a file to belinked, based on the file image 168, and identifies the image file C asa file to be liked, based on the file image 172.

The user may select the file images 168 and 172 from the stock area 170and superimpose the file images 168 and 172 on the image 178, byoperating the screen 48. Accordingly, the file images 168 and 172 arerepresented in the image 178, and the image 178 is transmitted as amessage content of the user to the automatic response AI.

As described above, when the document file A and the image file C areidentified as objects to be linked, the controller 26 may display theinformation of the files identified as objects to be linked, as amessage content of the automatic response AI on the screen 48. When theuser approves the message content (for example, when the user notifies amessage content answering “Yes”), information on linkage functionsexecutable by using the document file A and the image file C may bedisplayed in the display area 48 a as a message content 180 of theautomatic response AI. When the user designates a linkage function andgives an instruction to execute the linkage function by a messagecontent 182 in response to the message content 180, the controller 26executes the designated linkage function.

In addition, when the image 178 represents three or more file images,information on a linkage function executable by using the three or morefiles is notified. In addition, when the image 178 represents one fileimage, information on a function (a solo function) executable by using afile associated with the one file image is notified.

In addition, when a file image is selected from the stock area, theinformation on a linkage function or a solo function may be notifiedunder the control using the time limit described above. For example,when plural file images are taken out from the stock area andsuperimposed on the image 178 within the time limit, the specifying unit28 specifies a linkage function executable by using the plural files. Inthis case, the information on the linkage function is displayed in thedisplay area 48 a as a message content of the automatic response AI.Meanwhile, when one file image is taken out from the stock area andsuperimposed on the image 178 within the time limit, the specifying unit28 specifies a function (a solo function) executable by using the fileassociated with the file image. In this case, the information on thesolo function is displayed in the display area 48 a as a message contentof the automatic response AI.

In addition, as Modification 2 is applied, when plural file images aretransmitted as a message content of the user to the automatic responseAI, a composite image may be generated which represents a state wherethe plural file images are superimposed on each other (see FIG. 14A), ora state where the plural file images are adjacent and connected to eachother (see FIG. 14B). The composite image may be displayed in the stockarea 170. In addition, the composite image may be registered as ashortcut image in the terminal device 10.

In addition, as Modification 3 is applied, a linkage function to benotified may be changed according to an order of transmittinginformation of plural files to the automatic response AI. In this case,in the linkage function management table according to the fourthexemplary embodiment as well, access orders and priorities of pluralfile images are set as in the linkage function management tableillustrated in FIG. 15.

In addition, as Modification 4 is applied, different functions may beassigned to respective portions of a file image. In this case, whenportion images are transmitted as a message content of the user to theautomatic response AI, information on a linkage function executable byusing the functions associated with the portions is displayed. Inaddition, a combination of Modifications 3 and 4 may be applied to thefourth exemplary embodiment.

In addition, as Modification 5 is applied, information on a linkagefunction may be first input as a message content of the user by the userto the terminal device 10, and then, information on files to be linkedmay be input as a message content of the user to the terminal device 10.

In addition, as Modification 6 is applied, the notification of a linkagefunction may be controlled according to information obtained by theautomatic response AI. For example, the automatic response AI mayacquire information indicating a use history of a file, informationindicating an execution status of a device, information indicating anexecution status of software for implementing a function, information onthe user (information indicating an operation history or a behaviorhistory), and information indicating a user's schedule, by using thelearning function, and control a notification of a linkage function byusing the pieces of information.

In addition, as Modification 7 is applied, a function to be linked maybe recommended. For example, when the user transmits a file image as amessage content of the user to the automatic response AI, the specifyingunit 28 identifies the file associated with the file image and specifiesa function capable of executing the linkage function in combination withthe format of the identified file by referring to the linkage functionmanagement table. In addition, the specifying unit 28 may recommend afile based on a use history of a linkage function or recommend a filehaving a relatively high use frequency.

In addition, as Modification 8 is applied, the automatic response AI mayrecommend a function to be used for a linkage function based on theinformation shared with another automatic response AI.

In addition, as Modification 9 is applied, information of files to belinked may be input to the terminal device 10 by voice input.

In addition, in the fourth exemplary embodiment, when informationindicating names of plural files (file names) is transmitted as amessage content of the user to the automatic response AI, the specifyingunit 28 may specify a linkage function according to the plural filenames. To this end, a combination of the file names and a linkagefunction are associated with each other in the linkage functionmanagement table, and the specifying unit 28 specifies a linkagefunction associated with the combination of the plural file names byreferring to the linkage function management table.

In addition, the specifying unit 28 may specify a file to which acontent is to be added, based on a name of a file transmitted to theautomatic response AI. For example, when a name of a file (file name)transmitted to the automatic response AI includes a character stringsuggesting a file for collecting data (that is, a file assumed orpredicted to incorporate a content of other data), such as “ledger,”“collection,” “management,” “statistics,” “graph,” “table,” or “list,”the specifying unit 28 identifies the file as a to which a content is tobe added. In addition, the specifying unit 28 identifies a file having afile name including no character string for the collection or a filehaving a file name including a character string suggesting a file forproviding a content (material), as a file containing a content(material) to be added. For example, a file having a file name includinga character string such as “receipt,” “slip,” “personal data,” and“device data” does not correspond to a file for the collection and isdetermined to correspond to a file for providing a content (material).

For example, when the document files A1 and A2 which are files to belinked are transmitted as a message content of the user to the automaticresponse AI, in a case where the document file A1 has a file nameincluding the character string for the collection, and the document fileA2 has a file name including the character string for providing amaterial without including the character string for the collection, thespecifying unit 28 specifies a linkage function of performing, forexample, input, copying, and a statistical process from the documentfile A2 for providing a material to the document A1 for the collection,as a preferential linkage function. The controller 26 displays thepreferential linkage function on the screen 48 in preference to otherlinkage functions.

In addition, the combination of objects to be linked is not limited tothe combinations in the first to fourth exemplary embodiments. Forexample, when a combination of at least two of a device, a function(including a function executed by a device and a function executed bysoftware), a file, or software is selected as a combination of objectsto be linked, a notification of a linkage function executable by usingthe combination may be controlled. For example, when a device image anda file image are transmitted as a message content of the user to theautomatic response AI, the specifying unit 28 may specify a linkagefunction executable by using the device associated with the device imageand the file associated with the file image, and the controller 26 maydisplay information on a linkage function as a message content of theautomatic response AI on the screen 48 or output as voice information,as a notification of the information on the linkage function. As aspecific example, when a device image associated with the multifunctionmachine B and a file image associated with a document file aretransmitted as a message content of the user to the automatic responseAI, the specifying unit 28 specifies a linkage function executable byusing the multifunction machine B and the document file (for example, afunction of printing the document file by the multifunction machine B ora function of faxing the document file by the multifunction machine B).The controller 26 controls a notification of the information on thelinkage function. Similarly, when a function image associated withsoftware and a file image are transmitted as a message content of theuser to the automatic response AI, the specifying unit 28 may specify alinkage function executable by using the function associated with thefunction image and the file associated with the file image, and thecontroller 26 may control a notification of the information on thelinkage function. As a specific example, when a function imageassociated with a document application and a file image associated witha document file are transmitted as a message content of the user to theautomatic response AI, the specifying unit 28 specifies a linkagefunction executable by using the document application and the documentfile (for example, a function of editing or outputting the document fileby the document application). In addition, when a device image, afunction image, and a file image are transmitted as a message content ofthe user to the automatic response AI, the specifying unit 28 mayspecify a linkage function executable by using the device associatedwith the device image, the function associated with the function image(for example, software), and the file associated with the file image,and the controller 26 may control a notification of the information onthe linkage function. As a specific example, when a device imageassociated with the multifunction machine B, a function image associatedwith a document application, and a file image associated with a documentfile are transmitted as a message content of the user to the automaticresponse AI, the specifying unit 28 specifies a linkage functionexecutable by using the multifunction machine B, the documentapplication, and the document file (for example, a function of editingthe document file by the document application, output the document fileto the multifunction machine B, and print or fax the document file bythe multifunction machine B). In the specific examples described above,the combinations are merely examples. A linkage function executable byany combination of a device, a function, a file, and software other thanthe combinations described above may be set. In addition, information ofa device, a function, a file, and software may be transmitted to theautomatic response AI by voice.

The first to fourth exemplary embodiments and Modifications 1 to 9described above may also be applied when an automatic execution processby robotics process automation (RPA) is set. For example, when pieces ofinformation on a device, a function, or a file are transmitted by asetting person to the automatic response AI, a combination of thetransmitted pieces of information and a linkage function associated withthe combination are saved as setting contents, and a process is executedaccording to the setting contents. As for the RPA (Robotics ProcessAutomation), for example, a routine process such as data input or a jobexecutable by a combination of plural pieces of application software, aprocess such as data collection or analysis, or a process of learningand making a determination based on data may be performed.

Fifth Exemplary Embodiment

Hereinafter, a device system according to a fifth exemplary embodimentwill be described. In the fifth exemplary embodiment, when the usergives an instruction to use a device (a use request) to the automaticreplay AI (for example, when the user transmits information of a deviceto the automatic response AI), display of the conversation partner whois making a reply is switched to display of the device, on an interfacefor making a conversation with the automatic response AI (for example,the screen 48). Hereinafter, the fifth exemplary embodiment will bedescribed in detail.

FIG. 36 illustrates the screen 48. As in FIG. 8, the screen 48 displaysthe image 50 associated with the automatic response AI, the image 52associated with the user, and information indicating a conversationbetween the user and the automatic response AI.

For example, a message content 184 that “How can I help you?” isdisplayed in the display area 48 a as a message content of the automaticresponse AI. Since the message content 184 is a message content made bythe automatic response AI, the controller 26 displays the image 50associated with the automatic response AI in the display area 48 a, anddisplays the message content 184 in the display area 48 a as a messagecontent associated with the image 50 of the automatic response AI. Inaddition, the automatic response AI does not depend on a device, and maybe a general-purpose automatic response AI.

When the user designates a device in response to the message content184, the specifying unit 28 (the specifying unit 28 implemented by theautomatic response AI) identifies the device. For example, it is assumedthat the user makes a message content 186 of “I want to usemultifunction machine B” as a message content indicating an instructionto use a device (a use request). The controller 26 displays the messagecontent 186 including the information of the multifunction machine B inthe display area 48 b as a message content of the user, and transmitsthe message content 186 to the automatic response AI. Accordingly, theinformation of the multifunction machine B is transmitted to theautomatic response AI. The message content 186 is displayed in thedisplay area 48 b as a message content associated with the image 52 ofthe user. The user may input a character string indicating the device asa message content of the user to the terminal device 10, or designate adevice image (for example, an external appearance image or an icon)associated with the device as a message content of the user as in thefirst exemplary embodiment described above, by operating the UI unit 20.When designating a device image, as illustrated in FIG. 8, the user mayselect the device image associated with the device to be used from thestock area 58, and move the device image to the position where a messagecontent of the user is displayed, so as to designate the device image asa message content of the user.

The specifying unit 28 specifies the function that the device designatedby the user has by referring to the device function management tableillustrated in FIG. 6. In the example described above, since themultifunction machine B is designated, the specifying unit 28 identifiesthe function that the multifunction machine B has.

When the user designates the device and the device is identified, thecontroller 26 switches the display of the conversation partner (forexample, the image associated with the conversation partner or thecharacter string indicating the conversation partner), from the displayof the general-purpose automatic response AI (for example, the imageassociated with the general-purpose automatic response AI or thecharacter string indicating the general-purpose automatic response AI)to display of the device designated by the user (for example, an imageassociated with the device or a character string indicating the device).

In addition, even though the display of the conversation partner isswitched from the display of the general-purpose automatic response AIto the display of the device, the conversation partner itself of theuser is the general-purpose automatic response AI. That is, a messagecontent of the conversation partner is created by the general-purposeautomatic response AI equipped in the terminal device 10.

In the example described above, since the multifunction machine B isdesignated as a device to be used, the controller 26 displays a deviceimage 188 associated with the multifunction machine B in the displayarea 48 a as an image associated with a conversation partner in place ofthe image 50 associated with the automatic response AI. The device image188 may be an image generated by capturing the multifunction machine B(for example, an external appearance image of the multifunction machineB) or an image emulating the multifunction machine B (for example, anicon). In addition, the controller 26 may display a character stringindicating the multifunction machine B in the display area 48 a as theinformation indicating the conversation partner in place of or alongwith the device image 188.

In addition, the controller 26 may display a device image representing astate of the device in the display area 48 a as an image associated witha conversation partner in place of the image 50 associated with theautomatic response AI. In the example described above, the controller 26displays the device image 188 representing the state of themultifunction machine B in the display area 48 a as an image associatedwith a conversation partner. The controller 26 acquires the informationindicating the state of the device to be used from the device, andcontrols the display of the device image representing the state. Forexample, when the device is performing a process, the controller 26displays a device image representing the state where the device isperforming a process, as an image associated with a conversationpartner. When the device is not performing a process, the controller 26displays a device image representing the state where the device is notperforming a process, as an image associated with a conversationpartner. When the device is in a sleep state, the controller 26 displaysa device image representing the sleep state as an image associated witha conversation partner. When the device is powered off, the controller26 displays a device image representing the powered-off state as animage associated with a conversation partner. Accordingly, the user mayvisually confirm the state of the device. For example, when themultifunction machine B is performing printing, the controller 26displays the device image 188 representing the state where themultifunction machine B is performing printing, in the display area 48 aas an image associated with a conversation partner. The imagerepresenting the state of the device may be created in advance andstored in the memory 24, or may be created by the controller 26. Inaddition, the controller 26 may display a character string indicatingthe state of the device in the display area 48 a in place of or alongwith the device image.

Further, the controller 26 displays a message content 190 that “Whichfunction do you want to use?” in the display area 48 a as a messagecontent of the multifunction machine B. The message content 190 itselfis created by the general-purpose automatic response AI. In order to actas if the multifunction machine B makes the message content 190, thecontroller 26 displays the message content 190 in the display area 48 aas a message content associated with the device image 188 of themultifunction machine B. The controller 26 may display a list offunctions that the multifunction machine B specified by the specifyingunit 28 has in the display area 48 a as a message content of themultifunction machine B. Since the multifunction machine B is a devicehaving, for example, image forming functions such as the printingfunction and the copying function, the controller 26 may display acharacter string or an image representing a list of the image formingfunctions in the display area 48 a as a message content of themultifunction machine B.

In response to the message content 190, the user inputs a processcontent that the user wants to execute by using the multifunctionmachine B, as a message content of the user to the terminal device 10.In the example illustrated in FIG. 36, a message content 192 that “Iwant to print file X” which indicates the process content is input as amessage content of the user to the terminal device 10.

The controller 26 specifies the process content requested by the user byanalyzing the message content 192 of the user, and causes themultifunction machine B to execute the process content. For example, thecontroller 26 transmits a printing job including the data of the file Xto the multifunction machine B. Upon receiving the printing job, themultifunction machine B prints the file X according to the printing job.In addition, the controller 26 may cause the UI unit 20 to display ascreen for setting printing conditions (for example, conditions such asthe number of copies, color printing, and black and white printing). Theprinting job includes the conditions set on the screen, and themultifunction machine B performs the printing according to theconditions.

When the printing of the file X is completed, the controller 26 displaya message content 194 indicating that the printing has been completed,in the display area 48 a as a message content of the multifunctionmachine B. At this time as well, the controller 26 displays the deviceimage 188 associated with the multifunction machine B in the displayarea 48 a, and displays the message content 194 in the display area 48 aas a message content associated with the device image 188 of themultifunction machine B. In addition, the message content 194 itself iscreated by the general-purpose automatic response AI.

When the process by the multifunction machine B is completed, thecontroller 26 displays the image 50 associated with the general-purposeautomatic response AI in the display area 48 a as an image associatedwith a conversation partner in place of the device image 188 associatedwith the multifunction machine B. Further, the controller 26 displays amessage content 196 that “How can I help you?” which indicates a commonquestion in the display area 48 a as a message content of the automaticresponse AI.

As described above, according to the fifth exemplary embodiment, when aninstruction to use a device (a use request) is given to the automaticresponse AI (for example, when information of a device is transmitted tothe automatic response AI), the display of the conversation partner whois making a reply is switched to the display of the device, on thescreen 48 for making a conversation with the automatic response AI.Accordingly, the user may visually confirm whether the user may give theinstruction to the device. For example, when the user transmitsinformation of a device to the automatic response AI in order to givethe instruction to the device, an image associated with the device isdisplayed as an image associated with a conversation partner.Accordingly, the user may visually confirm whether the instruction hasbeen actually given to the device.

(Modification 10)

Hereinafter, Modification 10 will be described. In Modification 10, whenan instruction to use a device is assigned as a message content of theuser to the automatic response AI, the display of the conversationpartner is switched from the display of the general-purpose automaticresponse AI to display of the device, and the automatic response AI (forexample, a program) which is the conversation partner of the user isswitched from the general-purpose automatic response AI to an automaticresponse AI equipped in the device.

In the example described above, when the multifunction machine B isequipped with an automatic response AI (when a program of an automaticresponse AI is installed in the multifunction machine B), the controller26 switches the automatic response AI which is the conversation partnerof the user from the general-purpose automatic response AI equipped inthe terminal device 10 (the program of the automatic response AIinstalled in the terminal device 10) to the automatic response AIequipped in the multifunction machine B.

In this case, the controller 26 transmits information indicating amessage content of the user to the multifunction machine B via thecommunication path N. The automatic response AI equipped in themultifunction machine B receives the information indicating the messagecontent of the user that has been transmitted from the terminal device10, analyzes the message content of the user, and creates a responsesuch as a reply to the message content. The automatic response AI of themultifunction machine B transmits information indicating a messagecontent including the response to the terminal device 10. For example,as illustrated in FIG. 36, message contents 190 and 194 are created bythe automatic response AI equipped in the multifunction machine B, andthe information indicating the message contents 190 and 194 istransmitted from the multifunction machine B to the terminal device 10.The controller 26 displays the message contents transmitted from themultifunction machine B (for example, the message contents 190 and 194)in the display area 48 a as message contents of the multifunctionmachine B in association with the device image 188 of the multifunctionmachine B.

As illustrated in FIG. 36, when the process by the multifunction machineB is completed, the controller 26 switches the image associated with theconversation partner from the device image 188 associated with themultifunction machine B to the image 50 associated with thegeneral-purpose automatic response AI, and further, switches theautomatic response AI which is the conversation partner of the user fromthe automatic response AI equipped in the multifunction machine B to thegeneral-purpose automatic response AI equipped in the terminal device10.

According to Modification 10, when a device is equipped with anautomatic response AI, a conversation between the automatic response AIand the user is conducted, and thus, the processing load of theautomatic response AI equipped in the terminal device 10 is reduced. Inaddition, in a case where the automatic response AI equipped in thedevice has a high control ability or a high information collectionability, as compared with the general-purpose automatic response AI,when the automatic response AI equipped in the device which serves asthe conversation partner of the user makes a conversation with the user,more useful information may be provided to the user.

(Modification 11)

Hereinafter, Modification 11 will be described with reference to FIG.37. In Modification 11, when an instruction to use a portion of a device(a use request) is given to the automatic response AI (for example, wheninformation of a portion of a device is transmitted to the automaticresponse AI), the display of the conversation partner which is making areply is switched to display of the portion. Hereinafter, Modification11 will be described in detail.

FIG. 37 illustrates the screen 48. As in FIG. 8, the screen 48 displaysthe image 50 associated with the automatic response AI, the image 52associated with the user, and information indicating a conversationbetween the user and the automatic response AI.

As in the example illustrated in FIG. 36, a message content 198 that“How can I help you?” is displayed in the display area 48 a as a messagecontent of the automatic response AI, and in response to the messagecontent 198, a message content 200 that “I want to use multifunctionmachine B” which indicates an instruction to use a device is displayedas a message content of the user in the display area 48 b. Since themessage content 200 including the information of the multifunctionmachine B is transmitted to the automatic response AI, the imageassociated with the conversation partner is switched from the image 50associated with the general-purpose automatic response AI to the deviceimage 188 associated with the multifunction machine B, as in the fifthexemplary embodiment described above. Further, a message content 202that “Which function do you want to use?” is displayed in the displayarea 48 a as a message content of the multifunction machine B, and amessage content 204 that “I want to print file X” is displayed as amessage content of the user in the display area 48 b.

The specifying unit 28 specifies a process content requested by the userby analyzing the message content 204 of the user, and causes themultifunction machine B to execute the process content. As in the fifthexemplary embodiment described above, the controller 26 transmits theprinting job including the data of the file X to the multifunctionmachine B, and causes the multifunction machine B to execute theprinting job.

Further, the controller 26 switches the display of the conversationpartner to display of a portion of the device for executing the process.Since the printing function is assigned to the “main body portion” ofthe multifunction machine B, the controller 26 switches the display ofthe conversation partner to display of the “main body portion” of themultifunction machine B. For example, the controller 26 displays aportion image 206 associated with the “main body portion” of themultifunction machine B in the display area 48 a as an image associatedwith a conversation partner in place of the device image 188 associatedwith the multifunction machine B. In addition, when the message content204 indicating the process content is transmitted to the automaticresponse AI, it is regarded that an instruction to use the portion ofthe device for executing the process (a use request) has been given tothe automatic response AI. That is, it is regarded that the informationof the portion of the device has been transmitted to the automaticresponse AI. In the example described above, since the message content204 includes the process content that “I want to print,” it is regardedthat the information of the “main body portion” of the multifunctionmachine B to which the printing function is assigned has beentransmitted to the automatic response AI.

In addition, the controller 26 may display a portion image representinga state of a portion of a device in the display area 48 a as an imageassociated with a conversation partner in place of the image 50associated with the automatic response AI. In the example describedabove, the controller 26 displays the portion image 206 representing thestate of the “main body portion” of the multifunction machine B in thedisplay area 48 a as an image associated with a conversation partner.The controller 26 acquires the information indicating the state of eachportion of the device to be used from the device, and controls thedisplay of the portion image representing the state. For example, whenthe portion of the device is performing a process, the controller 26displays a portion image representing the state where the portion isperforming a process, as an image associated with a conversationpartner. When the portion of the device is not performing a process, thecontroller 26 displays a portion image representing the state where theportion is not performing a process, as an image associated with aconversation partner. When the portion of the device is in the sleepstate, the controller 26 displays a portion image representing the sleepstate as an image associated with a conversation partner. When thedevice is powered off, the controller 26 displays a portion imagerepresenting the powered-off state as an image associated with aconversation partner. Accordingly, the user may visually confirm thestate of the portion of the device. For example, when the “main bodyportion” of the multifunction machine B is performing printing, thecontroller 26 displays the portion image 206 representing the statewhere the “main body portion” of the multifunction machine B isperforming the printing, in the display area 48 a as an image associatedwith a conversation partner. The image representing the state of theportion of the device may be created in advance and stored in the memory24, or may be created by the controller 26. In addition, the controller26 may display a character string indicating the state of the portion ofthe device in the display area 48 a in place of or along with theportion image.

When the printing of the file X is completed, the controller 26 displaysa message content 208 indicating that the printing has been completed,in the display area 48 a as a message content of the “main body portion”of the multifunction machine B. At this time as well, the controller 26displays the portion image 206 associated with the “main body portion”of the multifunction machine B in the display area 48 a, and displaysthe message content 208 in the display area 48 a as a message contentassociated with the portion image 206 of the “main body portion” of themultifunction machine B.

When the process by the multifunction machine B is completed, thecontroller 26 displays the image 50 associated with the general-purposeautomatic response AI in the display area 48 a as an image associatedwith a conversation partner in place of the portion image 206 associatedwith the “main body portion” of the multifunction machine B. Further,the controller 26 displays a message content 210 that “How can I helpyou?” which indicates a common question in the display area 48 a as amessage content of the automatic response AI.

As another example, when information of a portion of a device (forexample, a portion image associated with a portion or a character stringindicating a portion) is transmitted as the message content 200 of theuser to the automatic response AI, the controller 26 may switch thedisplay of the conversation partner from the display of the automaticresponse AI to display of the portion. For example, when a messagecontent that “I want to use the main body portion (or printing portion)of multifunction machine B” is transmitted as the message content 200 ofthe user to the automatic response AI, the controller 26 may switch theimage associated with the conversation partner from the image 50associated with the automatic response AI to the portion image 206associated with the “main body portion” of the multifunction machine B.

As described above, according to Modification 11, when an instruction touse a portion of a device (a use request) is given to the automaticresponse AI (for example, when information of a portion of a device istransmitted to the automatic response AI), the display of theconversation partner is switched to the display of the portion of thedevice, on the screen 48 for making a conversation with the automaticresponse AI. Accordingly, the user may visually confirm whether the usermay give the instruction to the portion of the device.

In addition, when the portion of the device is displayed as aconversation partner, the conversation partner of the user may be thegeneral-purpose automatic response AI equipped in the terminal device 10or may be switched from the general-purpose automatic response AI to theautomatic response AI equipped in the device or portion.

(Modification 12)

Hereinafter, Modification 12 will be described with reference to FIG.38. In Modification 12, when the user gives an instruction to use pluraldevices (a use request) to the automatic response AI (for example, wheninformation of plural devices is transmitted as information of objectsto be linked to the automatic response AI), the display of theconversation partner who is making a reply is switched from the displayof the general-purpose automatic response AI to display of the pluraldevices. Hereinafter, Modification 12 will be described in detail.

FIG. 38 illustrates the screen 48. As in FIG. 8, the screen 48 displaysthe image 50 associated with the automatic response AI, the image 52associated with the user, and information indicating a conversationbetween the user and the automatic response AI.

As in the example illustrated in FIG. 36, a message content 212 that“How can I help you?” is displayed in the display area 48 a as a messagecontent of the automatic response AI.

When the user designates plural devices as devices to be linked inresponse to the message content 212, the specifying unit 28 (thespecifying unit 28 implemented by the automatic response AI) identifiesthe plural devices as devices to be linked. For example, it is assumedthat the user makes a message content 214 of “I want to usemultifunction machine B and PC A” as a message content indicating aninstruction to use plural devices (a use request). The controller 26displays the message content 214 including the information of themultifunction machine B and the PC A in the display area 48 b as amessage content of the user, and transmits the message content 214 tothe automatic response AI. Accordingly, the information of themultifunction machine B and the PC A is transmitted as information ofdevices to be linked to the automatic response AI. The user may inputcharacter strings indicating the plural devices as a message content ofthe user to the terminal device 10, or designate device images (forexample, external appearance images or icons) associated with thedevices as a message content of the user as in the first exemplaryembodiment described above, by operating the UI unit 20. Whendesignating device images, as illustrated in FIG. 8, the user may selectthe plural device images from the stock area 58 and move the pluraldevice images to the position where a message content of the user isdisplayed, so as to designate the plural device images as a messagecontent of the user.

The specifying unit 28 specifies linkage functions executable by usingthe plural devices designated by the user, by referring to, for example,the linkage function management table illustrated in FIG. 7. In theexample described above, since the multifunction machine B and the PC Aare designated, the “scanning transmitting function” and the “printingfunction” are specified as the linkage functions executable by using themultifunction machine B and the PC A (see FIG. 7).

When the user designates the plural devices, and the plural devices areidentified, the controller 26 switches the display of the conversationpartner from the display of the general-purpose automatic response AI todisplay of the plural devices designated by the user.

In the example described above, since the multifunction machine B andthe PC A are designated as devices to be linked, the controller 26displays the device image 188 associated with the multifunction machineB and the device image 216 associated with the PC A in the display area48 a as images associated with conversation partners in place of theimage 50 associated with the automatic response AI. The device image 216may be an image generated by capturing the PC A (for example, anexternal appearance image of the PC A) or an image emulating the PC A(for example, an icon). In addition, the controller 26 may displaycharacter strings indicating the multifunction machine B and the PC A inthe display area 48 a as information indicating the conversationpartners in place of or along with the device images 188 and 216.

Further, the controller 26 displays a message content 218 that “Whichfunction do you want to use?” in the display area 48 a as a messagecontent of the multifunction machine B and the PC A. The message content218 itself is created by the general-purpose automatic response AI. Inorder to act as if the multifunction machine B and the PC A make themessage content 218, the controller 26 displays the message content 218in the display area 48 a as a message content associated with the deviceimage 188 of the multifunction machine B and the device image 216 of thePC A. The controller 26 may display a list of linkage functionsspecified by the specifying unit 28 in the display area 48 a as amessage content of the multifunction machine B and the PC A. Since thelinkage functions executable by using the multifunction machine B andthe PC A are the “scanning transmitting function” and the “printingfunction” (see FIG. 7), the controller 26 may display character stringsor images indicating the “scanning transmitting function” and the“printing function” in the display area 48 a as a message content of themultifunction machine B and the PC A.

In response to the message content 218 described above, the user inputsa linkage function that the user wants to execute, as a message contentof the user to the terminal device 10. In the example illustrated inFIG. 38, a message content 220 that “I want to transmit scanned data toPC A” which indicates the scanning transmitting function is input as amessage content of the user to the terminal device 10.

The controller 26 specifies the linkage function requested by the userby analyzing the message content 220 of the user, and causes themultifunction machine B and the PC A to execute the linkage function.The process for executing the linkage function is the same as describedin the first exemplary embodiment described above.

When the scanning transmitting is completed, the controller 26 displaysa message content 222 indicating that the scanning transmitting has beencompleted, in the display area 48 a as a message content of themultifunction machine B and the PC A. At this time as well, thecontroller 26 displays the device image 188 associated with themultifunction machine B and the device image 216 associated with the PCA in the display area 48 a, and displays the message content 222 in thedisplay area 48 a as a message content associated with the device image188 of the multifunction machine B and the device image 216 of the PC A.In addition, the message content 222 itself is created by thegeneral-purpose automatic response AI.

When the process by the multifunction machine B and the PC A iscompleted, the controller 26 displays the image 50 associated with thegeneral-purpose automatic response AI in the display area 48 a as animage associated with a conversation partner in place of the deviceimage 188 associated with the multifunction machine B and the deviceimage 216 associated with the PC A. Further, the controller 26 displaysa message content 224 that “How can I help you?” which indicates acommon question, in the display area 48 a as a message content of theautomatic response AI.

As described above, according to Modification 12, when an instruction touse (a use request of) plural devices to be linked is given to theautomatic response AI (for example, when information of plural devicesto be linked is transmitted to the automatic response AI), the displayof the conversation partner is switched to the display of the pluraldevices, on the screen 48 for making a conversation with the automaticresponse AI. Accordingly, the user may visually confirm whether the usermay give the instruction the plural devices. For example, when the usertransmits information of plural devices to the automatic response AI inorder to give the instruction to the plural devices, the imagesassociated with the respective devices are displayed as imagesassociated with conversation partners. Accordingly, the user mayvisually confirm whether the instruction has been actually given to theplural devices.

In addition, when the plural devices are displayed as conversationpartners, the conversation partner of the user may be thegeneral-purpose automatic response AI equipped in the terminal device 10or may be switched from the general-purpose automatic response AI to theautomatic response AIs equipped in the plural devices. For example, whenthe multifunction machine B is equipped with an automatic response AI,the automatic response AI equipped in the multifunction machine B mayfunction as a conversation partner. When the PC A is equipped with anautomatic response AI, the automatic response AI equipped in the PC Amay function as a conversation partner. When both the multifunctionmachine B and the PC A are equipped with automatic response AIs, theautomatic response AI equipped in any one of the multifunction machine Band the PC A may function as a conversation partner.

In the example illustrated in FIG. 38, information of two devices istransmitted to the automatic response AI. However, information of threeor more devices (for example, three or more device images or characterstrings indicating three or more devices) may be transmitted to theautomatic response AI. In this case, the three or more device images aredisplayed in the display area 48 a as images associated withconversation partners in place of the image 50 associated with theautomatic response AI.

In addition, Modifications 11 and 12 may be combined with each other.That is, when plural portions are designated as portions to be linked,and information of the plural portions is transmitted to the automaticresponse AI, the controller 26 may switch the display of theconversation partner to display of the plural portions. In addition, theplural portions may be designated from the same device or pluraldevices.

(Modification 13)

Hereinafter, Modification 13 will be described with reference to FIGS.39A to 39E and FIGS. 40A to 40E. FIGS. 39A to 39E and FIGS. 40A to 40Eillustrate images associated with conversation partners of the user.Here, it is assumed that information of plural devices to be linked (forexample, the multifunction machine B and the PC A) is transmitted as aconversation content of the user to the automatic response AI.

FIG. 39A illustrates images in a basic form. As in Modification 12described above, when the information of the multifunction machine B andthe PC A is transmitted as a message content of the user to theautomatic response AI, the device image 188 associated with themultifunction machine B and the device image 216 associated with the PCA are displayed in the display area 48 a as images associated withconversation partners.

When the device images 188 and 216 are displayed as images associatedwith conversation partners as illustrated in FIG. 39A, and in thisstate, a message content 226 is displayed in the display area 48 a as amessage content of the multifunction machine B as illustrated in FIG.39B, the controller 26 displays the device image 188 associated with themultifunction machine B in the display area 48 a to be larger than thedevice image 216 associated with the PC A. For example, when the PC A isnot performing a process included in a linkage function, and themultifunction machine B is performing a process included in the linkagefunction, the information on the process is displayed in the displayarea 48 a as a message content of the multifunction machine B. In thiscase, the device image 188 is displayed larger than the device image216. Accordingly, the user may visually confirm the device which isperforming a process included in a linkage function. When both of themultifunction machine B and the PC A are performing a process includedin a linkage function, the device images 188 and 216 are displayed inthe same size as illustrated in FIG. 39A.

Meanwhile, as illustrated in FIG. 39C, when a message content 228 isdisplayed in the display area 48 a as a message content of the PC A, thecontroller 26 displays the device image 216 in the display area 48 a tobe larger than the device image 188. For example, when the PC A isperforming a process included in a linkage function, and themultifunction machine B is not performing a process included in thelinkage function, the information on the process is displayed in thedisplay area 48 a as a message content of the PC A.

In the example illustrated in FIGS. 38 and 39A, the device images 188and 216 are displayed to be arranged laterally side by side. However, asillustrated in FIG. 39D, the device images 188 and 216 may be displayedto be arranged vertically side by side. In addition, the device images188 and 216 may be arranged and displayed in other forms.

In addition, as illustrated in FIG. 39E, when a message content 230 isdisplayed in the display area 48 a as a message content of themultifunction machine B, the device image 188 associated with themultifunction machine B may be displayed larger than the device image216 associated with the PC A. When a message content is displayed in thedisplay area 48 a as a message content of the PC A, the device image 216is displayed larger than the device image 188.

In addition, the order of displaying a device image may be changedaccording to an order of messages of the user. For example, when theuser makes messages in order of the “multifunction machine B” and the“PC A”, the device images 188 and 216 may be displayed in an order ofthe device image 188 associated with the multifunction machine B and thedevice image 216 associated with the PC A. For example, a firstdesignated device (for example, the multifunction machine B) may bedisplayed at the left or upper side, and a second designated device (forexample, the PC A) may be displayed at the right or lower side. Inaddition, the displaying order may not follow the designating order, andmay be determined by an order of identifying or registering a device. Inaddition, a device image of a device which is making a message (a deviceassociated with a message content) may be displayed at a predeterminedposition (for example, at the left or upper side). Accordingly, the usermay visually confirm a device which is making a message (for example, adevice which is performing a process).

FIGS. 40A to 40E illustrate another example of the display. For example,in a case where Modifications 11 and 12 are combined with each other,when information of plural portions to be linked is transmitted as amessage content of the user to the automatic response AI, the controller26 switches the display of the conversation partner to display of theplural portions. FIG. 40A illustrates an example of the display. Forexample, when the “main body portion” of the multifunction machine B andthe “display unit” of the PC A are designated as portions to be linked,and information of the portions is transmitted as a message content ofthe user to the automatic response AI, the portion image 206 associatedwith the “main body” of the multifunction machine B and the portionimage 232 associated with the “display portion” of the PC A aredisplayed in the display area 48 a as images associated withconversation partners. In addition, when the plural portions aredisplayed as conversation partners, any one of the display examplesillustrated FIGS. 39A to 39E may be applied.

In addition, when a device is performing a process, the controller 26may display a portion image associated with the portion of the devicewhich is performing a process, in the display area 48 a as an imageassociated with a conversation partner. For example, when the “main bodyportion” of the multifunction machine B is performing printing, thecontroller 26 displays the portion image 206 associated with the “mainbody portion” in the display area 48 a as an image associated with aconversation partner as illustrated in FIG. 40B. Similarly, when the“reading unit” of the multifunction machine B is performing scanning,the controller 26 displays the portion image associated with the“reading portion” in the display area 48 a as an image associated with aconversation partner. Accordingly, the user may visually confirm theportion which is performing a process. In addition, the controller 26may display a portion image associated with a portion which isperforming a process (for example, the printing job) instructed by theuser on the screen 48 (for example, the “main body portion” of themultifunction machine B), in the display area 48 a as an imageassociated with a conversation partner. Accordingly, the user mayvisually confirm the portion which is performing the process instructedby the user.

In addition, as illustrated in FIG. 40C, an image 50 a representing theautomatic response AI itself and a background image 50 b representingthe background of the automatic response AI may be displayed in theimage 50 associated with the general-purpose automatic response AI. Wheninformation of a device (for example, the multifunction machine B) istransmitted to the automatic response AI, the entire image 50 may bereplaced with the device image 188 as illustrated in FIG. 40D. Asanother example, as illustrated in FIG. 40E, the image 50 may bereplaced with the device image 188 representing an image 188 arepresenting the multifunction machine B itself and the background image50 b. That is, the background image 50 b may be displayed as it is.

Sixth Exemplary Embodiment

Hereinafter, a device system according to a sixth exemplary embodimentwill be described. In the sixth exemplary embodiment, when the usertransmits an instruction to use a function (a use request) to theautomatic response AI (for example, when the user transmits informationof a function to the automatic response AI), the display of theconversation partner who is making a reply is switched to display of thefunction, on an interface for making a conversation with the automaticresponse AI (for example, the screen 48). Hereinafter, the sixthexemplary embodiment will be described in detail.

FIG. 41 illustrates the screen 48. As in FIG. 8, the screen 48 displaysthe image 50 associated with the automatic response AI, the image 52associated with the user, and information indicating a conversationbetween the user and the automatic response AI.

For example, a message content 234 that “How can I help you?” isdisplayed as a message content of the automatic response AI on thescreen 48. Since the message content 234 is a message content made bythe automatic response AI, the controller 26 displays the image 50associated with the automatic response AI in the display area 48 a, anddisplays the message content 234 in the display area 48 a as a messagecontent associated with the image 50 of the automatic response AI. Inaddition, the automatic response AI does not depend on a device, and maybe a general-purpose automatic response AI.

When the user designates a function in response to the message content234, the specifying unit 28 (the specifying unit 28 implemented by theautomatic response AI) identifies the function. For example, it isassumed that the user makes a message content 236 of “I want to usepassword setting function” as a message content indicating aninstruction to use the function (a use request). The controller 26displays the message content 236 including the information of thepassword setting function in the display area 48 b as a message contentof the user, and transmits the message content 236 to the automaticresponse AI. Accordingly, the information of the password settingfunction is transmitted to the automatic response AI. The messagecontent 236 is displayed in the display area 48 b as a message contentassociated with the image 52 of the user. The user may input a characterstring indicating the function as a message content of the user to theterminal device 10, or designate a function image (for example, an imagesuch as an icon) associated with the function as a message content ofthe user as in the second exemplary embodiment described above, byoperating the UI unit 20. When designating a function image, asillustrated in FIG. 27, the user may select the function imageassociated with the function to be used from the stock area 124 and movethe function image to the position where a message content of the useris displayed, so as to designate the function image as a message contentof the user.

When the user designates the function and the function is identified,the controller 26 switches the display of the conversation partner (forexample, the image associated with the conversation partner or thecharacter string indicating the conversation partner) from the displayof the general-purpose automatic response AI (for example, the imageassociated with the general-purpose automatic response AI or thecharacter string indicating the general-purpose automatic response AI)to the display of the function designated by the user (for example, theimage associated with the function or the character string indicatingthe function).

In addition, even though the display of the conversation partner isswitched from the display of the general-purpose automatic response AIto the display of the function, the conversation partner itself of theuser is the general-purpose automatic response AI. That is, a messagecontent of the conversation partner is created by the general-purposeautomatic response AI equipped in the terminal device 10.

In the example described above, since the password setting function isdesignated as a function to be used, the controller 26 displays afunction image 238 associated with the password setting function in thedisplay area 48 a as an image associated with a conversation partner inplace of the image 50 associated with the automatic response AI. Thefunction image 238 is, for example, an icon. In addition, the controller26 may display a character string indicating the password settingfunction in the display area 48 a as information indicating aconversation partner in place of or along with the function image 238.

In addition, the controller 26 may display a function image representinga state of the function (for example, a state of software or a devicefor implementing the function) in the display area 48 a as an imageassociated with a conversation partner in place of the image 50associated with the automatic response AI. In the example describedabove, the controller 26 displays the function image 238 representingthe state of password setting software in the display area 48 a as animage associated with a conversation partner. The controller 26 acquiresthe information indicating the state of the software or the device forimplementing the function from the software or device, and controls thedisplay of the function image representing the state. For example, whenthe software or device for implementing the function is performing aprocess, the controller 26 displays a function image representing thestate where the software or device is performing a process as an imageassociated with a conversation partner. When the software or device forimplementing the function is not performing a process, the controller 26displays a function image representing the state where the software ordevice is not performing a process as an image associated with aconversation partner. When the device for implementing the function isin the sleep state, the controller 26 displays a function imagerepresenting the sleep state as an image associated with a conversationpartner. When the software for implementing the function is in a freezestate, the controller 26 displays a function image representing thefreeze state as an image associated with a conversation partner. Whenthe device for implementing the function is powered off, the controller26 displays a function image representing the powered-off state as animage associated with a conversation partner. Accordingly, the user mayvisually confirm the state of the function. For example, when thepassword setting software is setting a password for another data, thecontroller 26 displays the function image 238 representing the statewhere the password setting software is performing the password settingprocess, in the display area 48 a as an image associated with aconversation partner. The image representing the state of the device maybe created in advance and stored in the memory 24, or may be created bythe controller 26. In addition, the controller 26 may display acharacter string indicating the state of the function in the displayarea 48 a in place of or along with the device image.

Further, the controller 26 displays a message content 240 that “Forwhich file do you want to set a password?” in the display area 48 a as amessage content of the password setting function. The message content240 itself is created by the general-purpose automatic response AI. Inorder to act as if the password setting function makes the messagecontent 240, the controller 26 displays the message content 240 in thedisplay area 48 a as a message content associated with the functionimage 238 of the password setting function.

In response to the message content 240, the user designates a file to besubjected to the password setting. In the example illustrated in FIG.41, a message content 242 that “File X” is input as a message content ofthe user to the terminal device 10.

The controller 26 specifies the process content requested by the user byanalyzing the message content 242 of the user, and causes the passwordsetting software to execute the process content. For example, thecontroller 26 activates the password setting software, and sets apassword for the file X designated by the user by the password settingsoftware. In addition, the controller 26 may cause the UI unit 20 todisplay a screen for setting the password.

When the password setting is completed, the controller 26 displays amessage content 244 indicating that the password setting has beencompleted, in the display area 48 a as a message content of the passwordsetting function. At this time as well, the controller 26 displays thefunction image 238 associated with the password setting function in thedisplay area 48 a, and displays the message content 244 in the displayarea 48 a as a message content associated with the function image 238 ofthe password setting function. In addition, the message content 244itself is created by the general-purpose automatic response AI.

When the process by the password setting function is completed, thecontroller 26 displays the image 50 associated with the general-purposeautomatic response AI in the display area 48 a as an image associatedwith a conversation partner in place of the function image 238associated with the password setting function. Further, the controller26 displays a message content 246 that “How can I help you?” whichindicates a common question, in the display area 48 a as a messagecontent of the automatic response AI.

As described above, according to the sixth exemplary embodiment, when aninstruction to use a function (a use request) is given to the automaticresponse AI (for example, when information of a function is transmittedto the automatic response AI), the display of the conversation partnerwho is making a reply is switched to the display of the function, on thescreen 48 for making a conversation with the automatic response AI.Accordingly, the user may visually confirm whether the user may give theinstruction to the function. For example, when the user transmits theinformation of the function to the automatic response AI in order togive the instruction to the function, the image associated with thefunction is displayed as an image associated with a conversationpartner. Accordingly, the user may visually confirm whether theinstruction has been actually given to the function.

Modification 10 may be applied to the sixth exemplary embodiment. Forexample, when a program of an automatic response AI is assigned to eachfunction, and the program of each automatic response AI is installed inthe terminal device 10, the controller 26 may switch the display of theconversation partner from the display of the general-purpose automaticresponse AI to the display of the function, and switch the automaticresponse AI which is the conversation partner of the user from thegeneral-purpose automatic response AI to the automatic response AIassigned to the function.

Modification 11 may be applied to the sixth exemplary embodiment. Forexample, in a case where a function is further assigned to each portionin a function image, when an image associated with the portion istransmitted as a message content of the user to the automatic responseAI, the image associated with the portion may be displayed in thedisplay area 48 a as an image associated with a conversation partner.

Modification 12 may be applied to the sixth exemplary embodiment. Forexample, when the user gives an instruction to use plural functions (ause request) to the automatic response AI (for example, when the usertransmits information of plural functions as information of objects tobe linked to the automatic response AI), the display of the conversationpartner who is making a reply is switched from the display of thegeneral-purpose automatic response AI to display of the pluralfunctions. Hereinafter, the process in a case where Modification 12 isapplied to the sixth exemplary embodiment will be described withreference to FIG. 42.

FIG. 42 illustrates the screen 48. As in FIG. 8, the screen 48 displaysthe image 50 associated with the automatic response AI, the image 52associated with the user, and information indicating a conversationbetween the user and the automatic response AI.

As in the example illustrated in FIG. 41, a message content 248 that“How can I help you?” is displayed in the display area 48 a as a messagecontent of the automatic response AI.

When the user designates plural functions as functions to be linked inresponse to the message content 248, the specifying unit 28 (thespecifying unit 28 implemented by the automatic response AI) identifiesthe plural functions as functions to be linked. For example, it isassumed that the user makes a message content 250 of “I want to usepassword setting function and data transmitting function” as a messagecontent indicating an instruction to use the plural functions (a userequest). The controller 26 displays the message content 250 includingthe information of the password setting function and the datatransmitting function in the display area 48 b as a message content ofthe user, and transmits the message content 250 as information ofobjects to be linked to the automatic response AI. Accordingly, theinformation of the password setting function and the data transmittingfunction is transmitted as information of functions to be linked to theautomatic response AI. The user may input character strings indicatingthe plural functions as a message content of the user to the terminaldevice 10, or designate function images (for example, icons) associatedwith the functions as a message content of the user as in the secondexemplary embodiment described above, by operating the UI unit 20. Whendesignating function images, as illustrated in FIG. 27, the user mayselect the plural function images from the stock area 124 and move theplural function images to the position where a message content of theuser is displayed, so as to designate the plural function images as amessage content of the user.

The specifying unit 28 specifies a linkage function executable by usingthe plural functions designated by the user, for example, by referringto the linkage function management table illustrated in FIG. 26. In theexample described above, since the password setting function and thedata transmitting function are designated, for example, the “function ofsetting password for data and transmitting the data” is specified as alinkage function.

When the user designates the plural functions and the plural functionsare identified, the controller 26 switches the display of theconversation partner from the display of the general-purpose automaticresponse AI to the display of the plural functions designated by theuser.

In the example described above, since the password setting function andthe data transmitting function are designated as functions to be linked,the controller 26 display the function image 238 associated with thepassword setting function and a function image 252 associated with thedata transmitting function in the display area 48 a as images associatedwith conversation partners in place of the image 50 associated with theautomatic response AI. In addition, the controller 26 may displaycharacter strings indicating the password setting function and the datatransmitting function in the display area 48 a as information indicatingconversation partners in place of or along with the function images 238and 252.

Further, the controller 26 displays a message content 254 that “Whichfile do you want to process?” in the display area 48 a as a messagecontent of the password setting function and the data transmittingfunction. The message content 254 itself is created by thegeneral-purpose automatic response AI. In order to act as if thepassword setting function and the data transmitting function make themessage content 254, the controller 26 displays the message content 254in the display area 48 a as a message content associated with thefunction image 238 of the password setting function and the functionimage 252 of the data transmitting function. The controller 26 maydisplay a list of linkage functions specified by the specifying unit 28in the display area 48 a as a message content of the password settingfunction and the data transmitting function.

In response to the message content 254, the user designates a file to beprocessed. In the example illustrated in FIG. 42, a message content 256that “File X” is input as a message content of the user to the terminaldevice 10.

The controller 26 specifies the process content requested by the user byanalyzing the message content 256 of the user, and causes the passwordsetting software and the data transmitting software to execute theprocess content. For example, the controller 26 activates the passwordsetting software and the data transmitting software, sets a password forthe file X designated by the user by using the password settingsoftware, and transmits the file X set with the password to atransmission destination (for example, a transmission destinationdesignated by the user or a predetermined transmission destination) byusing the data transmitting software. In addition, the controller 26 maycause the UI unit 20 to display a screen for setting a password or ascreen for setting a transmission destination.

When the execution of the linkage function is completed, the controller26 displays a message content 258 indicating that the execution of thelinkage function has been completed, in the display area 48 a as amessage content of the password setting function and the datatransmitting function. At this time as well, the controller 26 displaysthe function image 238 associated with the password setting function andthe function image 252 associated with the data transmitting function inthe display area 48 a, and displays the message content 258 in thedisplay area 48 a as a message content associated with the functionimage 238 of the password setting function and the function image 252 ofthe data transmitting function. In addition, the message content 258itself is created by the general-purpose automatic response AI.

When the execution of the linkage function is completed, the controller26 display the image 50 associated with the general-purpose automaticresponse AI in the display area 48 a as an image associated with aconversation partner in place of the function image 238 associated withthe password setting function and the function image 252 associated withthe data transmitting function. Further, the controller 26 displays amessage content 260 that “How can I help you?” which indicates a commonquestion, in the display area 48 a as a message content of the automaticresponse AI.

As described above, in a case where Modification 12 is applied to thesixth exemplary embodiment, when an instruction to use plural functionsto be linked (a use request) is assigned to the automatic response AI(for example, when information of plural functions to be linked istransmitted as a conversation content to the automatic response AI), thedisplay of the conversation partner is switched to the display of theplural functions, on the screen 48 for making a conversation with theautomatic response AI. Accordingly, the user may visually confirmwhether the user may give the instruction to the plural functions. Forexample, when the user transmits information of the plural functions tothe automatic response AI in order to give the instruction to the pluralfunctions, the images associated with the respective functions aredisplayed as images associated with conversation partners. Accordingly,the user may visually confirm whether the instruction has been actuallygiven to the plural functions.

In addition, when the plural functions are displayed as conversationpartners, the conversation partner of the user may be thegeneral-purpose automatic response AI or may be switched from thegeneral-purpose automatic response AI to the automatic response AIsequipped in the respective functions.

In the example illustrated in FIG. 42, information of two functions istransmitted to the automatic response AI. However, information of threeor more functions (for example, three or more function images orcharacter strings indicating three or more functions) may be transmittedto the automatic response AI. In this case, the three or more functionimages are displayed in the display area 48 a as images associated withconversation partners in place of the image 50 associated with theautomatic response AI.

In addition, a combination of Modifications 11 and 12 may be applied tothe sixth exemplary embodiment. That is, in a case where a function isfurther assigned to each portion in a function image, when imagesassociated with plural portions are transmitted as a message content ofthe user to the automatic response AI, the images of the plural portionsmay be displayed in the display area 48 a as images associated withconversation partners.

In addition, Modification 13 may be applied to the sixth exemplaryembodiment. That is, in the sixth exemplary embodiment as well, eachdevice image may be displayed according to the display examplesillustrated in FIGS. 39A to 39E and FIGS. 40A to 40E.

Seventh Exemplary Embodiment

Hereinafter, a device system according to a seventh exemplary embodimentwill be described. In the seventh exemplary embodiment, when the usergives an instruction to use a device and a function (a use request) tothe automatic response AI (for example, when the user transmitsinformation of a device and a function to the automatic response AI),the display of the conversation partner who is making a reply isswitched to display of the device and the function, on an interface formaking a conversation with the automatic response AI (for example, thescreen 48). Hereinafter, the seventh exemplary embodiment will bedescribed in detail.

FIG. 43 illustrates the screen 48. As in FIG. 8, the screen 48 displaysthe image 50 associated with the automatic response AI, the image 52associated with the user, and information indicating a conversationbetween the user and the automatic response AI.

For example, a message content 262 that “How can I help you?” isdisplayed as a message content of the automatic response AI on thescreen 48. Since the message content 262 is a message content made bythe automatic response AI, the controller 26 displays the image 50associated with the automatic response AI in the display area 48 a, anddisplays the message content 262 in the display area 48 a as a messagecontent associated with the image 50 of the automatic response AI. Inaddition, the automatic response AI does not depend on a device, and maybe a general-purpose automatic response AI.

When the user designates a device and a function as objects to be linkedin response to the message content 262, the specifying unit 28 (thespecifying unit 28 implemented by the automatic response AI) identifiesthe device and the function as objects to be linked. For example, it isassumed that the user makes a message content 264 of “I want to usemultifunction machine B and password setting function” as a messagecontent indicating an instruction to use a device and a function (a userequest). The controller 26 displays the message content 264 includingthe information of the multifunction machine B and the password settingfunction in the display area 48 b as a message content of the user, andtransmits the message content 264 to the automatic response AI.Accordingly, the information of the multifunction machine B and thepassword setting function is transmitted as information of objects to belinked to the automatic response AI. The message content 264 isdisplayed in the display area 48 b as a message content associated withthe image 52 of the user. The user may input character stringsindicating the device and the function as a message content of the userto the terminal device 10, or designate a device image associated withthe device and a function image associated with the function as amessage content of the user as in the third exemplary embodimentdescribed above, by operating the UI unit 20. When designating a deviceimage, as illustrated in FIG. 8, the user may select the device imageassociated with the device to be linked from the stock area 58 and movethe device image to the position where a message content of the user isdisplayed, so as to designate the device image as a message content ofthe user. Similarly, when designating a function image, as illustratedin FIG. 27, the user may select the function image associated with thefunction to be used from the stock area 124 and move the function imageto the position where a message content of the user is displayed, so asto designate the function image as a message content of the user.

The specifying unit 28 specifies a linkage function executable by usingthe device and the function designated by the user, by referring to, forexample, the linkage function management table illustrated in FIG. 30.In the example described above, since the multifunction machine B andthe password setting function are designated, for example, a “functionof setting password for data generated by scanning by multifunctionmachine B” is specified as a linkage function.

When the user designates the device and the function as objects to belinked and the device and the function are identified as objects to belinked, the controller 26 switches the display of the conversationpartner from the display of the general-purpose automatic response AI todisplay of the device and the function designated by the user.

In addition, even though the display of the conversation partner isswitched from the display of the general-purpose automatic response AIto the display of the device and the function, the conversation partneritself of the user is the general-purpose automatic response AI. Thatis, a message content of the conversation partner is created by thegeneral-purpose automatic response AI equipped in the terminal device10.

In the example described above, since the multifunction machine B andthe password setting function are designated as objects to be linked,the controller 26 displays the device image 188 associated with themultifunction machine B and the function image 238 associated with thepassword setting function in the display area 48 a as images associatedwith conversation partners in place of the image 50 associated with theautomatic response AI. In addition, the controller 26 may displaycharacter strings indicating the multifunction machine B and thepassword setting function in the display area 48 a as informationindicating conversation partners in place of or along with the deviceimage 188 and the function image 238.

Further, the controller 26 display a message content 266 that “Whichfunction do you want to use?” in the display area 48 a as a messagecontent of the multifunction machine B and the password settingfunction. The message content 266 itself is created by thegeneral-purpose automatic response AI. In order to act as if themultifunction machine B and the password setting function make themessage content 266, the controller 26 displays the message content 266in the display area 48 a as a message content associated with the deviceimage 188 of the multifunction machine B and the function image 238 ofthe password setting function. The controller 26 may display a list oflinkage functions specified by the specifying unit 28 in the displayarea 48 a as a message content of the multifunction machine B and thepassword setting function.

In response to the message content 266 described above, the user inputsa linkage function that the user wants to execute, as a message contentof the user to the terminal device 10. In the example illustrated inFIG. 43, a message content 268 that “I want to set password for scanneddata” which indicates a linkage function is input as a message contentof the user to the terminal device 10.

The controller 26 specifies the linkage function requested by the userby analyzing the message content 268 of the user, and causes themultifunction machine B and the password setting function (the passwordsetting software) to execute the linkage function. For example, thecontroller 26 transmits a scanning job to the multifunction machine Band causes the multifunction machine B to perform the scanning. Datagenerated by the scanning is transmitted from the multifunction machineB to the terminal device 10. Further, the controller 26 activates thepassword setting software and sets a password for the data by thepassword setting software.

When the execution of the linkage function is completed, the controller26 displays a message content 270 indicating that the execution of thelinkage function has been completed, in the display area 48 a as amessage content of the multifunction machine B and the password settingfunction. At this time as well, the controller 26 displays the deviceimage 188 associated with the multifunction machine B and the functionimage 238 associated with the password setting function in the displayarea 48 a, and displays the message content 270 in the display area 48 aas a message content associated with the device image 188 of themultifunction machine B and the function image 238 of the passwordsetting function. In addition, the message content 270 itself is createdby the general-purpose automatic response AI.

When the execution of the linkage function is completed, the controller26 displays the image 50 associated with the general-purpose automaticresponse AI in the display area 48 a as an image associated with aconversation partner in place of the device image 188 associated withthe multifunction machine B and the function image 238 associated withthe password setting function. Further, the controller 26 displays amessage content 272 that “How can I help you?” which indicates a commonquestion in the display area 48 a as a message content of the automaticresponse AI.

As described above, according to the seventh exemplary embodiment, whenan instruction to use a device and a function (a use request) is givento the automatic response AI (for example, when information of a deviceand a function is transmitted to the automatic response AI), the displayof the conversation partner is switched to the display of the device andthe function, on the screen 48 for making a conversation with theautomatic response AI. Accordingly, the user may visually confirmwhether the user may give the instruction to the device and thefunction. For example, when the user transmits information of a deviceand a function to the automatic response AI in order to give theinstruction to the device and the function, the images associated withthe device and the function are displayed as images associated withconversation partners. Accordingly, the user may visually confirmwhether the instruction has been actually given to the device and thefunction.

In the example illustrated in FIG. 43, information of the device and thefunction which total two is transmitted to the automatic response AI.However, information of a total of three or more devices and functionsmay be transmitted to the automatic response AI. In this case, the totalthree or more device images and function images are displayed in thedisplay area 48 a as images associated with conversation partners inplace of the image 50 associated with the automatic response AI.

Modification 10 may be applied to the seventh exemplary embodiment. Forexample, when the program of the automatic response AI is assigned toeach of a device and a function, the controller 26 may switch thedisplay of the conversation partner from the display of thegeneral-purpose automatic response AI to the display of the device andthe function, and switch the automatic response AI which is theconversation partner of the user from the general-purpose automaticresponse AI to the automatic response AIs assigned to the device and thefunction.

Modification 11 may be applied to the seventh exemplary embodiment. Forexample, when information of a portion of a device is transmitted as amessage content of the user to the automatic response AI, an imageassociated with the portion may be displayed as an image associated witha conversation partner. In addition, in a case where a function isfurther assigned to each portion in a function image, when an imageassociated with the portion is transmitted as a message content of theuser to the automatic response AI, the image associated with the portionmay be displayed in the display area 48 a as an image associated with aconversation partner.

In addition, Modification 13 may be applied to the seventh exemplaryembodiment. That is, in the seventh exemplary embodiment as well, adevice image or a function image may be displayed according to thedisplay examples in FIGS. 39A to 39E and FIGS. 40A to 40E.

Eighth Exemplary Embodiment

Hereinafter, a device system according to an eighth exemplary embodimentwill be described. In the eighth exemplary embodiment, when the usergives an instruction to use a file (a use request) to the automaticresponse AI (for example, when information of a file is transmitted tothe automatic response AI), the display of the conversation partner whois making a reply is switched to display of the file, on an interfacefor making a conversation with the automatic response AI (for example,the screen 48). Hereinafter, the eighth exemplary embodiment will bedescribed in detail.

FIG. 44 illustrates the screen 48. As in FIG. 8, the screen 48 displaysthe image 50 associated with the automatic response AI, the image 52associated with the user, and information indicating a conversationbetween the user and the automatic response AI.

For example, a message content 274 that “How can I help you?” isdisplayed in the display area 48 a as a message content of the automaticresponse AI. Since the message content 274 is a message content made bythe automatic response AI, the controller 26 displays the image 50associated with the automatic response AI in the display area 48 a, anddisplays the message content 274 in the display area 48 a as a messagecontent associated with the image 50 of the automatic response AI. Inaddition, the automatic response AI does not depend on a device, and maybe a general-purpose automatic response AI.

When the user designates a file in response to the message content 274,the specifying unit 28 (the specifying unit 28 implemented by theautomatic response AI) identifies the file. For example, it is assumedthat the user makes a message content 276 of “I want to edit documentfile A” which indicates an instruction to use a file (a use request).The controller 26 displays the message content 276 including theinformation of the document file A in the display area 48 b as a messagecontent of the user, and transmits the message content 276 to theautomatic response AI. Accordingly, the information of the document fileA is transmitted to the automatic response AI. The message content 276is displayed in the display area 48 b as a message content associatedwith the image 52 of the user. The user may input a character stringindicating the file (for example, a file name) as a message content ofthe user to the terminal device 10, or designate a file image associatedwith the file (for example, an image such as an icon) as a messagecontent of the user as in the fourth exemplary embodiment describedabove, by operating the UI unit 20. When designating a file image, asillustrated in FIG. 34, the user may select the file image associatedwith the file to be used from the stock area 170 and move the file imageto the position where a message content of the user is displayed, so asto designate the file image as a message content of the user.

When the user designates the file and the file is identified, thecontroller 26 switches the display of the conversation partner from thedisplay of the general-purpose automatic response AI to display of thefile designated by the user.

In addition, even though the display of the conversation partner isswitched from the display of the general-purpose automatic response AIto the display of the file, the conversation partner itself of the useris the general-purpose automatic response AI. That is, a message contentof the conversation partner is created by the general-purpose automaticresponse AI equipped in the terminal device 10.

In the example described above, since the document file A is designatedas a file to be used, the controller 26 displays a file image 278associated with the document file A in the display area 48 a as an imageassociated with a conversation partner in place of the image 50associated with the automatic response AI. The file image 278 is, forexample, an icon. In addition, the controller 26 may display a characterstring indicating the document file A (for example, a file name) in thedisplay area 48 a as information indicating a conversation partner inplace of or along with the file image 278.

Further, the controller 26 displays a message content 280 that “Wouldyou like to activate document application?” in the display area 48 a asa message content of the document file A. The message content 280 itselfis created by the general-purpose automatic response AI. In order to actas if the document file A makes the message content 280, the controller26 displays the message content 280 in the display area 48 a as amessage content associated with the file image 278 of the document fileA.

When the user instructs to activate the document application as in amessage content 282 in response to the message content 280 above, thecontroller 26 activates the document application according to theinstruction. At this time, the controller 26 displays a message content284 indicating that the document application is to be activated and thedocument file A is to be opened (the document file A is to be displayedto be editable), in the display area 48 a as a message content of thedocument file A. At this time as well, the controller 26 displays thefile image 278 associated with the document file A in the display area48 a, and displays the message content 284 in the display area 48 a as amessage content associated with the file image 278 of the document fileA. In addition, the message content 284 itself is created by thegeneral-purpose automatic response AI.

Thereafter, the user edits the document file A by using the documentapplication. When the editing is completed, the controller 26 displaysthe image 50 associated with the general-purpose automatic response AIin the display area 48 a as an image associated with a conversationpartner in place of the file image 278 associated with the document fileA.

As described above, according to the eighth exemplary embodiment, whenan instruction to use a file (a use request) is given to the automaticresponse AI (for example, when information of a file is transmitted tothe automatic response AI), the display of the conversation partner isswitched to the display of the file, on the screen 48 for making aconversation with the automatic response AI. Accordingly, the user mayvisually confirm whether the process on the file may be performed.

In addition, the function image associated with the software (theapplication) for processing the file may be displayed in the displayarea 48 a as an image associated with a conversation partner along withthe file image associated with the file. In the example described above,the function image associated with the document application may bedisplayed in the display area 48 a along with the file image 278associated with the document file A.

Modification 10 may be applied to the eighth exemplary embodiment. Forexample, when the program of the automatic response AI is assigned toeach file, and the program of each automatic response AI is installed inthe terminal device 10, the controller 26 may switch the display of theconversation partner from the display of the general-purpose automaticresponse AI to the display of the file, and switch the automaticresponse AI which is the conversation partner of the user from thegeneral-purpose automatic response AI to the automatic response AIassigned to the file.

Modification 12 may be applied to the eighth exemplary embodiment. Forexample, when the user gives an instruction to use plural files (a userequest) to the automatic response AI (for example, when the usertransmits information of plural files as information of objects to belinked to the automatic response AI), the display of the conversationpartner who is making a reply is switched from the display of thegeneral-purpose automatic response AI to display of the plural files.Hereinafter, the process in a case where Modification 12 is applied tothe eighth exemplary embodiment will be described with reference to FIG.45.

FIG. 45 illustrates the screen 48. As in FIG. 8, the screen 48 displaysthe image 50 associated with the automatic response AI, the image 52associated with the user, and information indicating a conversationbetween the user and the automatic response AI.

As in the example illustrated in FIG. 44, a message content 286 that“How can I help you?” is displayed in the display area 48 a as a messagecontent of the automatic response AI.

When the user designates plural files as files to be linked in responseto the message content 286, the specifying unit 28 (the specifying unit28 implemented by the automatic response AI) identifies the plural filesas files to be linked. For example, it is assumed that the user makes amessage content 288 of “I want to edit document file A and image file C”which indicates an instruction to use plural files (a use request). Thecontroller 26 displays the message content 288 including the informationof the document file A and the image file C in the display area 48 b asa message content of the user, and transmits the message content 288 asinformation of objects to be linked to the automatic response AI.Accordingly, the information of the document file A and the image file Cis transmitted as information of the files to be linked to the automaticresponse AI. The user may input character strings indicating the pluralfiles (for example, file names) as a message content of the user to theterminal device 10, or designate file images associated with the files(for example, icons) as a message content of the user as in the fourthexemplary embodiment, by operating the UI unit 20. When designating fileimages, as illustrated in FIG. 34, the user may select the plural fileimages from the stock area 170 and move the plural file images to theposition where a message content of the user is displayed, so as todesignate the plural file images as a message content of the user.

The specifying unit 28 specifies a linkage function executable by usingthe plural files designated by the user by referring to, for example,the linkage function management table illustrated in FIG. 33. In theexample described above, since the document file A and the image file Care designated, for example, the “function of inserting an image into adocument” or the “function of superimposing a document on an image” isspecified as a linkage function.

When the user designates the plural files and the plural files areidentified, the controller 26 switches the display of the conversationpartner from the display of the general-purpose automatic response AI todisplay of the plural files designated by the user.

In the example described above, since the document file A and the imagefile C are designated as files to be linked, the controller 26 displaysthe file image 278 associated with the document file A and a file image290 associated with the image file C in the display area 48 a as imagesassociated with conversation partners in place of the image 50associated with the automatic response AI. In addition, the controller26 may display character strings indicating the document file A and theimage file C (for example, file names) in the display area 48 a asinformation indicating conversation partners in place of or along withthe file images 278 and 290.

Further, the controller 26 displays a message content 292 that “Whichfunction do you want to use?” in the display area 48 a as a messagecontent of the document file A and the image file C. The message content292 itself is created by the general-purpose automatic response AI. Inorder to act as if the document file A and the image file C make themessage content 292, the controller 26 displays the message content 292in the display area 48 a as a message content associated with the fileimage 278 of the document file A and the file image 290 of the imagefile C. The controller 26 may display a list of linkage functionsspecified by the specifying unit 28 in the display area 48 a as amessage content of the document file A and the image file C.

In response to the message content 292 described above, the user inputsa linkage function that the user wants to execute, as a message contentof the user to the terminal device 10. In the example illustrated inFIG. 45, a message content 294 that “I want to insert an image into adocument” which indicates a linkage function is input as a messagecontent of the user to the terminal device 10.

The controller 26 specifies the linkage function requested by the userby analyzing the message content 294 of the user, and controls executionof the linkage function. In the example described above, the controller26 activates the document application and the image application so as toinsert the image file C into the document file A.

When the execution of the linkage function is completed, the controller26 displays a message content 296 indicating that the execution of thelinkage function has been completed, in the display area 48 a as amessage content of the document file A and the image file C. At thistime as well, the controller 26 displays the file image 278 associatedwith the document file A and the file image 290 associated with theimage file C in the display area 48 a, and displays the message content296 in the display area 48 a as a message content associated with thefile image 278 of the document file A and the file image 290 of theimage file C. In addition, the message content 296 itself is created bythe general-purpose automatic response AI.

When the execution of the linkage function is completed, the controller26 displays the image 50 associated with the general-purpose automaticresponse AI in the display area 48 a as an image associated with aconversation partner in place of the file image 278 associated with thedocument file A and the file image 290 associated with the image file C.Further, the controller 26 displays a message content 298 that “How canI help you?” which indicates a common question in the display area 48 aas a message content of the automatic response AI.

In addition, the controller 26 may display a document editing screen andan image editing screen on the display unit of the UI unit 20 byactivating the document application and the image application. When thelinkage function described above is executed, the document file A in astate where the image file C is inserted into the document file A isdisplayed on the document editing screen. The user may edit the documentfile A or the image file C on the document editing screen. At the timepoint when the editing is ended or started, the controller 26 displaysthe image 50 associated with the general-purpose automatic response AIin the display area 48 a as an image associated with a conversationpartner in place of the file images 278 and 290.

In addition, the function image associated with the software (theapplication) for processing the file may be displayed in the displayarea 48 a as an image associated with a conversation partner along withthe file image associated with the file. In the example described above,a function image associated with the document application and a functionimage associated with the image application may be displayed in thedisplay area 48 a along with the file images 278 and 290.

As described above, in a case where Modification 12 is applied to theeighth exemplary embodiment, when an instruction to use plural files tobe linked (a use request) is given to the automatic response AI (forexample, when information of plural files to be linked is transmitted tothe automatic response AI), the display of the conversation partner isswitched to the display of the plural files, on the screen 48 for makinga conversation with the automatic response AI. Accordingly, the user mayvisually confirm whether the process on the plural files may beperformed.

In addition, when the plural files are displayed as conversationpartners, the conversation partner of the user may be thegeneral-purpose automatic response AI or may be switched from thegeneral-purpose automatic response AI to the automatic response AIsequipped in the respective files.

In the example illustrated in FIG. 45, information of two files istransmitted to the automatic response AI. However, information of threeor more files (for example, three or more file images or characterstrings indicating three or more files) may be transmitted to theautomatic response AI. In this case, the three or more file images aredisplayed in the display area 48 a as images associated withconversation partners in place of the image 50 associated with theautomatic response AI.

In addition, Modification 13 may be applied to the eighth exemplaryembodiment. That is, in the eighth exemplary embodiment as well, eachdevice image may be displayed according to the display examplesillustrated in FIGS. 39A to 39E and FIGS. 40A to 40E.

In addition, the combination of objects to be linked is not limited tothose in the fifth to eighth exemplary embodiments. For example, wheninformation of a combination of at least two of a device, a function(including a function executed by a device and a function executed bysoftware), a file, or software is transmitted as information of objectsto be linked to the automatic response AI, the display of thegeneral-purpose automatic response AI may be switched to display of thecombination.

For example, when information of a device and a file (for example, adevice image and a file image) is transmitted as a message content ofthe user to the automatic response AI, the controller 26 switches theimage associated with the conversation partner from the image 50associated with the general-purpose automatic response AI to the deviceimage and the file image. As a specific example, when the information ofthe multifunction machine B and the document file is transmitted as amessage content of the user to the automatic response AI, the controller26 switches the image associated with the conversation partner from theimage 50 associated with the general-purpose automatic response AI tothe device image associated with the multifunction machine B and thefile image associated with the document file.

Similarly, when information of software and a file (for example, afunction image associated with software and a file image) is transmittedas a message content of the user to the automatic response AI, thecontroller 26 switches the image associated with the conversationpartner from the image 50 associated with the general-purpose automaticresponse AI to the function image and the file image. As a specificexample, when the information of the document application and thedocument file is transmitted as a message content of the user to theautomatic response AI, the controller 26 switches the image associatedwith the conversation partner from the image 50 associated with thegeneral-purpose automatic response AI to the function image associatedwith the document application and the file image associated with thedocument file.

In addition, when information of a device, a function, and a file istransmitted as a message content of the user to the automatic responseAI, the controller 26 switches the image associated with theconversation partner from the image associated with the general-purposeautomatic response AI to the device image, the function image, and thefile image. As a specific example, when the information of themultifunction machine B, the document application, and the document fileis transmitted as a message content of the user to the automaticresponse AI, the controller 26 switches the image associated with theconversation partner from the image 50 associated with thegeneral-purpose automatic response AI to the device image associatedwith the multifunction machine B, the function image associated with thedocument application, and the file image associated with the documentfile.

The combinations in the specific examples described above are merelyexamples. When information of another combination of a device, afunction, a file, and software is transmitted to the automatic responseAI, the display of the general-purpose automatic response AI may beswitched to display of the combination.

Each of the terminal device 10, the devices 12 and 14, and the server 16is implemented by, for example, cooperation of hardware and software.Specifically, each of the terminal device 10, the devices 12 and 14, andthe server 16 includes one processor or plural processors such as a CPU(not illustrated). The function of each unit of the terminal device 10,the devices 12 and 14, and the server 16 is implemented in the mannerthat the one processor or the plural processors read and execute aprogram stored in a storage device (not illustrated). The program isstored in the storage device via a recording medium such as a CD or aDVD or a communication path such as a network. As another example, eachunit of the terminal device 10, the devices 12 and 14, and the server 16may be implemented by, for example, a hardware resource such as aprocessor, an electronic circuit, or an application specific integratedcircuit (ASIC). For the implementation, a device such as a memory may beused. As yet another example, each unit of the terminal device 10, thedevices 12 and 14, and the server 16 may be implemented by, for example,a digital signal processor (DSP) or a field programmable gate array(FPGA).

The foregoing description of the exemplary embodiments of the presentinvention has been provided for the purposes of illustration anddescription. It is not intended to be exhaustive or to limit theinvention to the precise forms disclosed. Obviously, many modificationsand variations will be apparent to practitioners skilled in the art. Theembodiments were chosen and described in order to best explain theprinciples of the invention and its practical applications, therebyenabling others skilled in the art to understand the invention forvarious embodiments and with the various modifications as are suited tothe particular use contemplated. It is intended that the scope of theinvention be defined by the following claims and their equivalents.

What is claimed is:
 1. An information processing device comprising: acontroller that controls a notification of a linkage function which isexecutable by using at least one object to be linked when a usertransmits information of the object to be linked to a conversationpartner on an interface for making a conversation with the conversationpartner.
 2. The information processing device according to claim 1,wherein the at least one object to be linked includes a plurality ofobjects to be linked, and when information of the plurality of objectsto be linked is transmitted to the conversation partner, the controllercontrols the notification of the linkage function which is executable byusing the plurality of objects to be linked.
 3. The informationprocessing device according to claim 2, wherein the controller changesthe notification of the linkage function according to an order oftransmitting the information of the plurality of objects to be linked.4. The information processing device according to claim 2, wherein whenimages representing the plurality of objects to be linked aretransmitted to the conversation partner, the controller controls thenotification of the linkage function executable by using the pluralityof objects to be linked.
 5. The information processing device accordingto claim 1, wherein the controller causes the interface to displayinformation of one or a plurality of candidates for the at least oneobject to be linked, and transmits information of an object to be linkedwhich is selected from the candidates, to the conversation partner. 6.The information processing device according to claim 1, wherein the atleast one object to be linked is a device, and when information of thedevice which corresponds to information of the at least one object to belinked is transmitted to the conversation partner, the controllercontrols the notification of the linkage function executable by usingthe device.
 7. The information processing device according to claim 6,wherein the information of the device is an image associated with thedevice.
 8. The information processing device according to claim 7,wherein the image associated with the device is an external appearanceimage of the device.
 9. The information processing device according toclaim 8, wherein the external appearance image is obtained by capturingthe device.
 10. The information processing device according to claim 9,wherein the controller transmits the captured external appearance imageto the conversation partner after the conversation partner makes aquestion about the device.
 11. The information processing deviceaccording to claim 6, wherein the at least one object to be linkedincludes portions of the device, and when information of the portionswhich corresponds to information of the at least one object to be linkedis transmitted to the conversation partner, the controller controls thenotification of the linkage function executable by using the portions.12. The information processing device according to claim 6, wherein asthe notification of the linkage function, the controller controls anotification of an interface configured to control the device in orderto execute the linkage function.
 13. The information processing deviceaccording to claim 1, wherein the at least one object to be linkedincludes functions, and when information of the functions whichcorresponds to information of the at least one object to be linked istransmitted to the conversation partner, the controller controls thenotification of the linkage function executable by using the functions.14. The information processing device according to claim 13, wherein theinformation of the functions are images associated with the functions.15. The information processing device according to claim 1, wherein theat least one object to be linked includes a device and a function, andwhen information of the device and the function which corresponds toinformation of the at least one object to be linked is transmitted tothe conversation partner, the controller controls the notification ofthe linkage function executable by using the device and the function.16. The information processing device according to claim 15, wherein theat least one object to be linked includes a portion of the device andthe function, and when information of the portion and the function whichcorresponds to information of the at least one object to be linked istransmitted to the conversation partner, the controller controls thenotification of the linkage function executable by using the portion andthe function.
 17. The information processing device according to claim1, wherein the at least one object to be linked includes files, and wheninformation of the files which corresponds to information of the atleast one object to be linked is transmitted to the conversationpartner, the controller controls the notification of the linkagefunction executable by using the files.
 18. The information processingdevice according to claim 17, wherein the controller changes thenotification of the linkage function according to formats of the files.19. The information processing device according to claim 1, wherein theat least one object to be linked includes pieces of software, and wheninformation of the pieces of software which corresponds to informationof the at least one object to be linked is transmitted to theconversation partner, the controller controls the notification of thelinkage function executable by using the pieces of software.
 20. Theinformation processing device according to claim 1, wherein the at leastone object to be linked is a combination of at least two of a device, afunction, a file, or software, and when information of the combinationwhich corresponds to information of the at least one object to be linkedis transmitted to the conversation partner, the controller controls thenotification of the linkage function executable by using thecombination.
 21. A non-transitory computer readable medium storing aprogram causing a computer to execute information processing, theinformation processing comprising: controlling a notification of alinkage function which is executable by using at least one object to belinked when a user transmits information of the object to be linked to aconversation partner on an interface for making a conversation with theconversation partner.